2024-03-29T10:27:54.406 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:27:54.406 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645/orig.config.yaml 2024-03-29T10:27:54.441 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:27:54.527 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622645' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDR2+Wh4EFdavoNAlwXZOvp3kKLwaLJcfAlicDL6P656byf9my2aWpeW5A3gmSLnaGRIoRC2nm50uiRzDUcvhsI= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApCkfzolq0sEtzqzhFSLQvasiZ2FY7OxW3B1Ce2+TbmTeFff2PlE1cXJlwWU7g+i40pHhoavGZC0srYa6M9OlY= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:27:54.528 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:27:54.528 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:27:54.528 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:27:54.530 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:27:54.554 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:27:54.555 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:27:54.555 INFO:teuthology.packaging:ref: None 2024-03-29T10:27:54.555 INFO:teuthology.packaging:tag: None 2024-03-29T10:27:54.555 INFO:teuthology.packaging:branch: main 2024-03-29T10:27:54.555 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:27:54.555 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:27:54.810 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:27:54.811 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:27:54.823 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:27:54.823 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:27:54.838 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:27:54.853 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:27:54.868 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:27:54.894 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:22:52.591831', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDR2+Wh4EFdavoNAlwXZOvp3kKLwaLJcfAlicDL6P656byf9my2aWpeW5A3gmSLnaGRIoRC2nm50uiRzDUcvhsI='} 2024-03-29T10:27:54.919 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:22:52.593074', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApCkfzolq0sEtzqzhFSLQvasiZ2FY7OxW3B1Ce2+TbmTeFff2PlE1cXJlwWU7g+i40pHhoavGZC0srYa6M9OlY='} 2024-03-29T10:27:54.919 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:27:54.931 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:27:54.931 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:27:54.931 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:27:55.013 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fda0c8370d0>, signals=[15]) 2024-03-29T10:27:55.013 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:27:55.025 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:27:55.025 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T10:27:55.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:27:55.119 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T10:27:55.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:27:55.208 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:27:55.215 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-03-29T10:27:55.233 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-03-29T10:27:55.233 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-03-29T10:27:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-03-29T10:27:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-03-29T10:27:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-03-29T10:27:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:27:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-03-29T10:27:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:27:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:27:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-03-29T10:27:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:27:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:27:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:27:55.290 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:27:55.290 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:27:55.290 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:27:55.290 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-03-29T10:27:55.312 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-03-29T10:27:55.330 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-03-29T10:27:55.330 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-03-29T10:27:55.387 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-03-29T10:27:55.387 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2024-03-29T10:27:55.387 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-03-29T10:27:55.387 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:27:55.387 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2024-03-29T10:27:55.387 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:27:55.387 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:27:55.388 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-03-29T10:27:55.388 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:27:55.388 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:27:55.388 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:27:55.388 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:27:55.388 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:27:55.388 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:27:55.388 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-03-29T10:27:55.404 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:27:55.416 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:27:55.428 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:27:55.428 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:27:55.431 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:27:55.446 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:27:55.458 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:27:55.458 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:27:55.485 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:27:55.703 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:27:55.757 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:27:55.774 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:27:55.774 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:27:55.774 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:27:55.775 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:27:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:27:55.791 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-03-29T10:27:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:27:55.793 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-29T10:27:55.848 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:27:55.848 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-361.el9.x86_64 2024-03-29T10:27:55.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-03-29T10:27:55.850 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:27:55.850 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.14.0-361.el9.x86_64 2024-03-29T10:27:55.850 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2024-03-29T10:27:57.914 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:27:57.968 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T10:27:59.531 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-29T10:27:59.568 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-29T10:28:00.546 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T10:28:01.575 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.0 MB 00:01 2024-03-29T10:28:02.972 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T10:28:06.093 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-03-29T10:28:07.198 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.1 MB 00:00 2024-03-29T10:28:08.723 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-29T10:28:10.710 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T10:28:15.526 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 1.0 MB/s | 6.1 MB 00:05 2024-03-29T10:28:15.641 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-29T10:28:16.910 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-03-29T10:28:17.923 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:28:17.923 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:28:18.005 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:28:18.022 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:28:18.022 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T10:28:18.022 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:28:18.022 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:28:18.022 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:28:18.022 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T10:28:18.022 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:28:18.022 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:28:18.023 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:28:18.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:28:18.023 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:28:18.023 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:28:18.023 INFO:teuthology.orchestra.run.smithi176.stdout:Install 4 Packages 2024-03-29T10:28:18.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:28:18.023 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 76 M 2024-03-29T10:28:18.023 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 123 M 2024-03-29T10:28:18.023 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:28:18.184 INFO:teuthology.orchestra.run.smithi176.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T10:28:18.439 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-29T10:28:18.676 INFO:teuthology.orchestra.run.smithi176.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-29T10:28:19.268 INFO:teuthology.orchestra.run.smithi176.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-29T10:28:19.884 INFO:teuthology.orchestra.run.smithi176.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-29T10:28:19.887 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:28:19.888 INFO:teuthology.orchestra.run.smithi176.stdout:Total 41 MB/s | 76 MB 00:01 2024-03-29T10:28:19.888 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:28:20.265 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:28:20.265 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:28:21.078 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:28:21.079 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:28:23.386 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T10:28:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:28:24.033 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:28:24.117 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:28:25.685 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:28:25.685 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:28:25.791 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:28:25.802 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:28:25.802 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 76 M 2024-03-29T10:28:25.803 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 123 M 2024-03-29T10:28:25.804 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:28:26.013 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 918 kB/s | 192 kB 00:00 2024-03-29T10:28:26.813 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 15 MB/s | 15 MB 00:01 2024-03-29T10:28:26.874 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:28:27.009 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:28:27.471 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-03-29T10:28:27.863 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T10:28:27.866 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:28:27.866 INFO:teuthology.orchestra.run.smithi114.stdout:Total 37 MB/s | 76 MB 00:02 2024-03-29T10:28:27.867 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:28:28.240 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:28:28.240 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:28:29.109 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:28:29.109 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:28:31.402 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:28:31.734 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:28:31.782 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:28:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:28:32.205 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:28:34.951 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:28:35.098 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:28:36.521 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:28:39.836 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:28:39.880 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:28:44.585 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:06.183 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:06.183 INFO:teuthology.orchestra.run.smithi176.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:29:06.183 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:29:06.232 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:06.581 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:06.581 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:29:06.581 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:29:06.581 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:29:06.993 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:06.993 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:29:06.993 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:29:06.993 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:06.993 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:29:06.993 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:29:06.993 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:29:06.993 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:29:06.993 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:29:07.311 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:29:07.455 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:07.455 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:29:07.455 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:29:07.455 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:29:07.455 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:29:07.456 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:29:07.456 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:29:07.456 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:29:07.536 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:07.536 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:29:07.536 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2024-03-29T10:29:07.570 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2024-03-29T10:29:07.570 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2024-03-29T10:29:07.570 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.06 2024-03-29T10:29:07.570 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 68.el9 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:Group : Unspecified 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 8264254 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : builder@centos.org 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:29:07.571 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2024-03-29T10:29:07.572 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2024-03-29T10:29:07.572 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:29:07.572 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:29:07.572 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:29:07.572 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:29:07.572 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2024-03-29T10:29:07.572 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:29:07.572 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:29:07.573 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:29:07.573 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:29:07.573 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:29:08.183 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-03-29T10:29:09.553 INFO:teuthology.orchestra.run.smithi176.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:29:09.589 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-03-29T10:29:09.591 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-03-29T10:29:09.607 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.21IK8hETgu 2024-03-29T10:29:09.607 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.21IK8hETgu 2024-03-29T10:29:09.675 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.21IK8hETgu 2024-03-29T10:29:09.811 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.21IK8hETgu is 6KB 2024-03-29T10:29:09.860 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.21IK8hETgu 2024-03-29T10:29:09.875 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:29:09.941 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:29:09.941 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:29:09.941 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:29:09.941 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:29:09.942 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:29:10.081 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-03-29T10:29:13.830 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:13.830 INFO:teuthology.orchestra.run.smithi114.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:29:13.830 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:29:13.873 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:14.261 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:14.261 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:29:14.261 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:29:14.262 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:29:14.689 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:14.689 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:29:14.689 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:29:14.689 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:14.689 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:29:14.689 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:29:14.689 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:29:14.689 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:29:14.690 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:29:15.005 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:29:15.100 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:15.100 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:29:15.100 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:29:15.100 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:29:15.100 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:29:15.100 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:29:15.100 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:29:15.101 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:29:15.180 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:15.180 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:29:15.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-03-29T10:29:15.216 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-03-29T10:29:15.216 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-03-29T10:29:15.216 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-03-29T10:29:15.216 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 68.el9 2024-03-29T10:29:15.216 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-03-29T10:29:15.216 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8264254 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-03-29T10:29:15.217 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:29:15.218 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:29:15.218 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:29:15.218 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:29:15.218 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-03-29T10:29:15.218 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:29:15.218 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:29:15.219 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:29:15.219 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:29:15.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:29:15.850 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-03-29T10:29:17.180 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:29:17.215 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-03-29T10:29:17.219 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-03-29T10:29:17.234 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.LF5piD9iMF 2024-03-29T10:29:17.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LF5piD9iMF 2024-03-29T10:29:17.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.LF5piD9iMF 2024-03-29T10:29:17.440 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.LF5piD9iMF is 6KB 2024-03-29T10:29:17.488 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.LF5piD9iMF 2024-03-29T10:29:17.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:29:17.566 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:29:17.567 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:29:17.567 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:29:17.567 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:29:17.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:29:17.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-03-29T10:29:40.106 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:29:40.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T10:29:40.108 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:29:47.723 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:29:47.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-29T10:29:47.724 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:30:14.559 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-29T10:30:23.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T10:30:23.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:30:41.951 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-29T10:30:47.731 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:30:53.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T10:30:53.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:30:54.332 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-29T10:30:54.732 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T10:30:54.732 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:30:54.732 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-29T10:30:54.746 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:30:54.747 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:30:54.747 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:30:54.747 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:30:55.748 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:30:55.748 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:30:55.834 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-03-29T10:30:55.887 DEBUG:teuthology.parallel:result is None 2024-03-29T10:30:56.732 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-29T10:30:56.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:30:57.153 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-03-29T10:30:57.602 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-29T10:30:57.602 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:30:57.602 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-03-29T10:30:57.616 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:30:57.617 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:30:57.617 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:30:57.617 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:30:58.618 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:30:58.618 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:30:58.686 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-03-29T10:30:58.736 DEBUG:teuthology.parallel:result is None 2024-03-29T10:30:58.736 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:30:58.745 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:30:58.745 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:30:58.747 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:30:58.774 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:30:58.781 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:30:58.791 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:30:58.791 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:30:58.804 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:30:58.847 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:30:58.854 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:30:58.854 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:30:58.865 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:30:58.906 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:30:58.919 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:30:58.923 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:30:58.943 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:30:58.945 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:30:58.953 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:30:58.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:30:58.965 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:30:59.013 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:30:59.022 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:30:59.023 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:30:59.031 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:30:59.058 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:30:59.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:30:59.206 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:30:59.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:30:59.289 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:30:59.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:30:59.359 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:30:59.398 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:30:59.449 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:30:59.528 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:30:59.566 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:30:59.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:30:59.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-03-29T10:30:59.635 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-03-29T10:30:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:30:59.738 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:31:00.249 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:31:00.257 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:31:00.257 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:31:00.268 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:31:00.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-03-29T10:31:00.336 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-03-29T10:31:00.337 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-03-29T10:31:00.377 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-03-29T10:31:00.379 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:31:00.379 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-03-29T10:31:00.397 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-03-29T10:31:00.398 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-03-29T10:31:00.442 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-03-29T10:31:00.442 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi176.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:31:00.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:31:00.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:31:00.469 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:31:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:31:00.508 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:31:00.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:31:00.641 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:31:00.680 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:31:00.690 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:31:00.709 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:31:00.711 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:35:12.683 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2024-03-29T10:35:12.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-29T10:35:12.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:35:12.820 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-03-29T10:35:12.906 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-29T10:35:12.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T10:35:12.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:35:12.997 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-29T10:35:13.086 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T10:35:13.086 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:35:13.100 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:35:13.100 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:35:13.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:35:13.108 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:35:13.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:35:13.136 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:35:13.153 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:35:13.156 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:35:13.172 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:35:13.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-03-29T10:35:13.196 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-03-29T10:35:13.202 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-03-29T10:35:13.213 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:35:13.215 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-03-29T10:35:13.229 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:35:13.231 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:35:13.248 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:35:13.288 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:35:13.292 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:35:13.292 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-03-29T10:35:13.292 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:13.292 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:13.292 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:13.292 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:13.303 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:35:13.305 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:35:13.306 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-03-29T10:35:13.306 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:13.306 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:13.306 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:13.306 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:13.311 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:35:13.320 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:35:13.320 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:35:13.320 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:35:13.320 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:35:13.325 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:35:13.325 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:35:13.326 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:35:13.326 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:35:13.328 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:35:13.328 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:35:13.473 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:35:13.473 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:35:13.576 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:35:13.576 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:35:13.664 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:35:13.664 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:35:13.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:35:13.684 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:35:13.684 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:35:13.684 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:35:13.692 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:35:13.692 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:35:13.712 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:35:13.712 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:35:13.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:35:13.779 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:35:13.840 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:35:13.860 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:35:13.906 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-03-29T10:35:13.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-03-29T10:35:13.926 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2024-03-29T10:35:13.928 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2024-03-29T10:35:14.154 INFO:teuthology.orchestra.run.smithi114.stdout:74 files removed 2024-03-29T10:35:14.169 INFO:teuthology.orchestra.run.smithi176.stdout:74 files removed 2024-03-29T10:35:14.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw 2024-03-29T10:35:14.196 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw 2024-03-29T10:35:14.952 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 181 kB/s | 82 kB 00:00 2024-03-29T10:35:14.967 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 169 kB/s | 82 kB 00:00 2024-03-29T10:35:15.277 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 36 kB/s | 10 kB 00:00 2024-03-29T10:35:15.618 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 17 kB/s | 10 kB 00:00 2024-03-29T10:35:15.711 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-29T10:35:16.210 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 3.4 kB/s | 1.9 kB 00:00 2024-03-29T10:35:16.405 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-29T10:35:16.745 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T10:35:18.031 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-29T10:35:18.437 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-29T10:35:19.146 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-29T10:35:20.645 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.0 MB 00:01 2024-03-29T10:35:26.280 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 3.1 MB/s | 19 MB 00:06 2024-03-29T10:35:27.037 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 3.5 MB/s | 19 MB 00:05 2024-03-29T10:35:30.963 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T10:35:32.224 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-03-29T10:35:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 16 kB 00:00 2024-03-29T10:35:33.658 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-03-29T10:35:34.593 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-29T10:35:39.492 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-29T10:35:42.161 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:35:42.165 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:35:42.166 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:35:42.167 INFO:teuthology.orchestra.run.smithi176.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:35:42.168 INFO:teuthology.orchestra.run.smithi176.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:35:42.168 INFO:teuthology.orchestra.run.smithi176.stdout:Installing weak dependencies: 2024-03-29T10:35:42.168 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:35:42.168 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:35:42.168 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:35:42.168 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:35:42.168 INFO:teuthology.orchestra.run.smithi176.stdout:Install 30 Packages 2024-03-29T10:35:42.168 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 2 Packages 2024-03-29T10:35:42.168 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:35:42.169 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 58 M 2024-03-29T10:35:42.169 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:35:44.051 INFO:teuthology.orchestra.run.smithi176.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.1 MB/s | 5.3 MB 00:01 2024-03-29T10:35:44.101 INFO:teuthology.orchestra.run.smithi176.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 503 kB/s | 25 kB 00:00 2024-03-29T10:35:44.252 INFO:teuthology.orchestra.run.smithi176.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.8 MB/s | 742 kB 00:00 2024-03-29T10:35:44.377 INFO:teuthology.orchestra.run.smithi176.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.9 MB/s | 498 kB 00:00 2024-03-29T10:35:44.574 INFO:teuthology.orchestra.run.smithi176.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 9.7 MB 00:02 2024-03-29T10:35:44.636 INFO:teuthology.orchestra.run.smithi176.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 679 kB/s | 45 kB 00:00 2024-03-29T10:35:44.712 INFO:teuthology.orchestra.run.smithi176.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.7 MB/s | 131 kB 00:00 2024-03-29T10:35:44.796 INFO:teuthology.orchestra.run.smithi176.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.9 MB/s | 162 kB 00:00 2024-03-29T10:35:44.887 INFO:teuthology.orchestra.run.smithi176.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.4 MB/s | 320 kB 00:00 2024-03-29T10:35:44.987 INFO:teuthology.orchestra.run.smithi176.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.9 MB/s | 298 kB 00:00 2024-03-29T10:35:45.079 INFO:teuthology.orchestra.run.smithi176.stdout:(11/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 7.3 MB/s | 5.1 MB 00:00 2024-03-29T10:35:45.096 INFO:teuthology.orchestra.run.smithi176.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 911 kB/s | 99 kB 00:00 2024-03-29T10:35:45.280 INFO:teuthology.orchestra.run.smithi176.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 869 kB/s | 159 kB 00:00 2024-03-29T10:35:45.305 INFO:teuthology.orchestra.run.smithi176.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 470 kB/s | 106 kB 00:00 2024-03-29T10:35:45.338 INFO:teuthology.orchestra.run.smithi176.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 754 kB/s | 44 kB 00:00 2024-03-29T10:35:45.422 INFO:teuthology.orchestra.run.smithi176.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 661 kB 00:00 2024-03-29T10:35:45.456 INFO:teuthology.orchestra.run.smithi176.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2024-03-29T10:35:45.481 INFO:teuthology.orchestra.run.smithi176.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 710 kB/s | 41 kB 00:00 2024-03-29T10:35:45.506 INFO:teuthology.orchestra.run.smithi176.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T10:35:45.531 INFO:teuthology.orchestra.run.smithi176.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-29T10:35:45.557 INFO:teuthology.orchestra.run.smithi176.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 924 kB/s | 46 kB 00:00 2024-03-29T10:35:45.882 INFO:teuthology.orchestra.run.smithi176.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 882 kB/s | 309 kB 00:00 2024-03-29T10:35:45.940 INFO:teuthology.orchestra.run.smithi176.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-03-29T10:35:45.999 INFO:teuthology.orchestra.run.smithi176.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-03-29T10:35:46.066 INFO:teuthology.orchestra.run.smithi176.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-03-29T10:35:46.091 INFO:teuthology.orchestra.run.smithi176.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 734 kB/s | 67 kB 00:00 2024-03-29T10:35:46.133 INFO:teuthology.orchestra.run.smithi176.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-29T10:35:46.159 INFO:teuthology.orchestra.run.smithi176.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-29T10:35:46.217 INFO:teuthology.orchestra.run.smithi176.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-29T10:35:46.543 INFO:teuthology.orchestra.run.smithi176.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 4.7 MB/s | 20 MB 00:04 2024-03-29T10:35:47.493 INFO:teuthology.orchestra.run.smithi176.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.5 MB/s | 3.4 MB 00:01 2024-03-29T10:35:47.543 INFO:teuthology.orchestra.run.smithi176.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 2.4 MB/s | 3.1 MB 00:01 2024-03-29T10:35:47.549 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:47.549 INFO:teuthology.orchestra.run.smithi176.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-29T10:35:47.550 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:35:47.586 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:35:47.587 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:35:48.060 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:35:48.060 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:35:49.502 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:35:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:35:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:35:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:35:49.646 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:35:49.662 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:35:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:35:49.910 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:35:50.227 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:35:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:35:50.373 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:35:50.537 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:35:50.584 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:35:50.647 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:35:50.675 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:35:50.719 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:35:50.786 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:35:50.811 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:35:50.875 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:35:50.910 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:35:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:35:51.061 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:35:51.102 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:35:51.134 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:35:51.164 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:35:51.196 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:35:51.521 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:35:51.551 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:35:51.624 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:35:51.655 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:35:51.693 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:35:52.157 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:35:52.187 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:35:52.258 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:35:52.317 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:35:52.348 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:35:54.507 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:35:54.560 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:35:55.115 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:35:55.147 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:35:55.225 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:35:55.225 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:35:55.225 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:35:55.225 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:35:55.256 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:36:04.692 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:36:04.693 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2024-03-29T10:36:04.693 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2024-03-29T10:36:04.693 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2024-03-29T10:36:04.693 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2024-03-29T10:36:04.694 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2024-03-29T10:36:04.694 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /root 2024-03-29T10:36:04.694 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2024-03-29T10:36:04.694 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:05.592 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:36:05.616 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:36:05.616 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:05.616 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:36:05.617 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:36:05.617 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:36:05.617 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:05.618 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:36:05.703 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:36:05.703 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:36:06.832 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:36:06.833 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:36:06.833 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:36:06.833 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:36:06.833 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:36:06.833 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:36:06.833 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:36:06.833 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:36:06.833 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:36:06.833 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:36:06.833 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:36:06.833 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:36:06.834 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:36:06.834 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:36:06.834 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:36:06.834 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:36:06.834 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:36:06.834 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:36:06.834 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:36:06.834 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:36:06.834 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:36:06.835 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:36:06.836 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:36:07.853 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:36:07.853 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:07.853 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2024-03-29T10:36:07.853 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.853 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.853 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:36:07.853 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:36:07.854 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.854 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.854 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.854 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.854 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:36:07.854 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:36:07.854 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:36:07.854 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.854 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:36:07.854 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:36:07.854 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.855 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:07.856 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:36:07.856 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:36:07.856 INFO:teuthology.orchestra.run.smithi176.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:36:07.856 INFO:teuthology.orchestra.run.smithi176.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:36:07.856 INFO:teuthology.orchestra.run.smithi176.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:36:07.856 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:07.856 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:36:08.118 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-test 2024-03-29T10:36:08.671 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:36:08.821 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:36:08.822 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:36:08.822 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T10:36:08.822 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:36:08.822 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:36:08.822 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:36:08.822 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T10:36:08.822 INFO:teuthology.orchestra.run.smithi176.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:36:08.822 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:36:08.822 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:36:08.823 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:36:08.823 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:36:08.823 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:08.823 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:36:08.823 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:36:08.823 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-03-29T10:36:08.823 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:08.824 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 48 M 2024-03-29T10:36:08.824 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 204 M 2024-03-29T10:36:08.824 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:36:09.081 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-29T10:36:09.131 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:36:09.165 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:36:09.290 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 412 kB/s | 163 kB 00:00 2024-03-29T10:36:09.324 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-29T10:36:12.341 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 14 MB/s | 47 MB 00:03 2024-03-29T10:36:12.343 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:12.344 INFO:teuthology.orchestra.run.smithi176.stdout:Total 14 MB/s | 48 MB 00:03 2024-03-29T10:36:12.358 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:36:12.379 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:36:12.379 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:36:12.757 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:36:12.757 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:36:13.324 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:36:13.391 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:36:13.431 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:36:13.471 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:36:13.516 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:36:13.544 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:36:18.002 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:36:18.076 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:36:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:36:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:36:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:36:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:36:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:36:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:36:20.879 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:36:20.879 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:20.879 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:36:20.879 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:20.879 INFO:teuthology.orchestra.run.smithi176.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:36:20.879 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:20.879 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:36:20.879 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:36:20.879 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:36:20.879 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:20.879 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:36:21.176 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph 2024-03-29T10:36:21.756 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:42 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:36:21.926 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:36:21.928 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-29T10:36:21.928 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T10:36:21.928 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-29T10:36:21.928 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:36:21.928 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:36:21.929 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:36:21.930 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout:Install 36 Packages 2024-03-29T10:36:21.931 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:21.932 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 30 M 2024-03-29T10:36:21.932 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 107 M 2024-03-29T10:36:21.932 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:36:22.476 INFO:teuthology.orchestra.run.smithi176.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-29T10:36:23.177 INFO:teuthology.orchestra.run.smithi176.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 2.1 MB 00:00 2024-03-29T10:36:23.219 INFO:teuthology.orchestra.run.smithi176.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 1.5 MB 00:00 2024-03-29T10:36:23.385 INFO:teuthology.orchestra.run.smithi176.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.5 MB/s | 247 kB 00:00 2024-03-29T10:36:23.522 INFO:teuthology.orchestra.run.smithi176.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T10:36:23.561 INFO:teuthology.orchestra.run.smithi176.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.6 MB/s | 144 kB 00:00 2024-03-29T10:36:23.586 INFO:teuthology.orchestra.run.smithi176.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T10:36:23.611 INFO:teuthology.orchestra.run.smithi176.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-29T10:36:23.637 INFO:teuthology.orchestra.run.smithi176.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-29T10:36:23.679 INFO:teuthology.orchestra.run.smithi176.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T10:36:23.746 INFO:teuthology.orchestra.run.smithi176.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T10:36:23.779 INFO:teuthology.orchestra.run.smithi176.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T10:36:23.805 INFO:teuthology.orchestra.run.smithi176.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 4.8 MB/s | 124 kB 00:00 2024-03-29T10:36:23.847 INFO:teuthology.orchestra.run.smithi176.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.3 MB/s | 187 kB 00:00 2024-03-29T10:36:24.014 INFO:teuthology.orchestra.run.smithi176.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 749 kB/s | 125 kB 00:00 2024-03-29T10:36:24.281 INFO:teuthology.orchestra.run.smithi176.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 84 kB/s | 22 kB 00:00 2024-03-29T10:36:24.423 INFO:teuthology.orchestra.run.smithi176.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-29T10:36:24.473 INFO:teuthology.orchestra.run.smithi176.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 867 kB/s | 43 kB 00:00 2024-03-29T10:36:24.523 INFO:teuthology.orchestra.run.smithi176.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-03-29T10:36:24.590 INFO:teuthology.orchestra.run.smithi176.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-29T10:36:24.632 INFO:teuthology.orchestra.run.smithi176.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-03-29T10:36:24.674 INFO:teuthology.orchestra.run.smithi176.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 425 kB/s | 18 kB 00:00 2024-03-29T10:36:24.716 INFO:teuthology.orchestra.run.smithi176.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 557 kB/s | 23 kB 00:00 2024-03-29T10:36:24.758 INFO:teuthology.orchestra.run.smithi176.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 466 kB/s | 19 kB 00:00 2024-03-29T10:36:24.817 INFO:teuthology.orchestra.run.smithi176.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 338 kB/s | 20 kB 00:00 2024-03-29T10:36:24.859 INFO:teuthology.orchestra.run.smithi176.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-29T10:36:24.901 INFO:teuthology.orchestra.run.smithi176.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.9 MB/s | 79 kB 00:00 2024-03-29T10:36:24.951 INFO:teuthology.orchestra.run.smithi176.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-29T10:36:24.993 INFO:teuthology.orchestra.run.smithi176.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2024-03-29T10:36:25.035 INFO:teuthology.orchestra.run.smithi176.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-29T10:36:25.077 INFO:teuthology.orchestra.run.smithi176.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 859 kB/s | 36 kB 00:00 2024-03-29T10:36:25.127 INFO:teuthology.orchestra.run.smithi176.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T10:36:25.186 INFO:teuthology.orchestra.run.smithi176.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-03-29T10:36:25.219 INFO:teuthology.orchestra.run.smithi176.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 599 kB/s | 20 kB 00:00 2024-03-29T10:36:27.003 INFO:teuthology.orchestra.run.smithi176.stdout:(35/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 4.4 MB/s | 17 MB 00:03 2024-03-29T10:36:27.078 INFO:teuthology.orchestra.run.smithi176.stdout:(36/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 4.7 MB 00:04 2024-03-29T10:36:27.082 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:27.082 INFO:teuthology.orchestra.run.smithi176.stdout:Total 5.8 MB/s | 30 MB 00:05 2024-03-29T10:36:27.107 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:36:27.149 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:36:27.149 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:36:27.563 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:36:27.564 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:36:28.914 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:36:28.972 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:36:29.007 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:36:29.043 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:36:29.113 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:36:29.150 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:36:29.183 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:36:29.274 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:36:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:36:29.385 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:36:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:36:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:36:29.549 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:36:29.598 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:36:29.653 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:36:29.837 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:36:29.887 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:36:29.917 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:36:29.952 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:36:29.982 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:36:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:36:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:36:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:36:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:36:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:36:30.226 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:36:30.316 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:36:30.357 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:36:30.429 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:36:30.565 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:36:30.682 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:36:30.856 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:36:30.887 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:36:30.913 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:36:30.913 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:30.913 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:36:30.913 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:36:30.914 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:36:30.914 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:32.338 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:36:32.365 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:36:32.365 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:32.365 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:36:32.365 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:36:32.365 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:36:32.365 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:32.805 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:36:32.828 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:36:32.828 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:32.829 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:36:32.829 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:36:32.829 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:36:32.829 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:33.031 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:36:33.054 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:36:33.054 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:33.054 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:36:33.055 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:36:33.055 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:36:33.055 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:33.137 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:36:34.853 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:36:34.854 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:36:34.854 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:36:34.854 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:36:34.854 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:36:34.854 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:36:34.854 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:36:34.854 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:36:34.854 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:36:34.854 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:36:34.854 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:36:34.854 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:36:34.855 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:36:34.857 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:36:34.857 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:36:34.857 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:36:34.857 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:36:34.857 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:36:34.857 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:36:34.857 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:36:34.857 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:36:34.857 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:36:34.857 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:36:35.955 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:36:35.956 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:36:35.957 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:35.958 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:36:36.322 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-base 2024-03-29T10:36:36.897 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:36:36.997 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:36:37.051 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:36:37.052 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T10:36:37.052 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:36:37.123 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install cephadm 2024-03-29T10:36:37.258 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 328 kB/s | 21 MB 01:04 2024-03-29T10:36:37.687 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:36:37.842 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:36:37.842 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:36:37.842 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T10:36:37.842 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:36:37.842 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:36:37.842 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:36:37.842 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:37.842 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:36:37.843 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:36:37.843 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T10:36:37.843 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:37.843 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 747 k 2024-03-29T10:36:37.843 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 754 k 2024-03-29T10:36:37.843 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:36:38.272 INFO:teuthology.orchestra.run.smithi176.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-29T10:36:38.272 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:38.272 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.7 MB/s | 747 kB 00:00 2024-03-29T10:36:38.273 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:36:38.278 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:36:38.278 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:36:38.283 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:36:38.284 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:36:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:36:38.755 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:36:38.827 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:36:39.199 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:36:39.518 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:36:39.519 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:39.519 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:36:39.519 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:36:39.519 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:39.519 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:36:39.756 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:36:40.329 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:36:40.484 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:36:40.485 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:36:40.485 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-29T10:36:40.485 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:36:40.485 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:36:40.486 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:36:40.486 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:40.486 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:36:40.486 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:36:40.486 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T10:36:40.486 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:40.486 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 145 k 2024-03-29T10:36:40.486 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 436 k 2024-03-29T10:36:40.486 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:36:40.704 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 671 kB/s | 145 kB 00:00 2024-03-29T10:36:40.704 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:40.704 INFO:teuthology.orchestra.run.smithi176.stdout:Total 667 kB/s | 145 kB 00:00 2024-03-29T10:36:40.704 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:36:40.712 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:36:40.712 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:36:40.764 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:36:40.764 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:36:40.905 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:36:40.962 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:36:40.987 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:36:40.987 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:40.987 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:36:40.987 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:42.045 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:36:42.045 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:42.046 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:36:42.046 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:42.046 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:42.046 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:36:42.191 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-03-29T10:36:42.308 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr 2024-03-29T10:36:42.884 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:36:42.982 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:36:43.035 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:36:43.036 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T10:36:43.036 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:36:43.101 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:36:43.657 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:36:43.807 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:36:43.808 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-29T10:36:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T10:36:43.808 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-29T10:36:43.808 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:36:43.809 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-29T10:36:43.810 INFO:teuthology.orchestra.run.smithi176.stdout:Install 12 Packages 2024-03-29T10:36:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:43.810 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 6.9 M 2024-03-29T10:36:43.810 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 26 M 2024-03-29T10:36:43.810 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:36:44.164 INFO:teuthology.orchestra.run.smithi176.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 76 kB/s | 15 kB 00:00 2024-03-29T10:36:44.231 INFO:teuthology.orchestra.run.smithi176.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 117 kB/s | 31 kB 00:00 2024-03-29T10:36:44.415 INFO:teuthology.orchestra.run.smithi176.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.9 MB/s | 1.8 MB 00:00 2024-03-29T10:36:44.448 INFO:teuthology.orchestra.run.smithi176.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-03-29T10:36:44.465 INFO:teuthology.orchestra.run.smithi176.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 238 kB 00:00 2024-03-29T10:36:44.724 INFO:teuthology.orchestra.run.smithi176.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-29T10:36:44.778 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:36:44.797 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:36:44.797 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T10:36:44.797 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:36:44.798 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse 2024-03-29T10:36:44.799 INFO:teuthology.orchestra.run.smithi114.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:36:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:36:44.801 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:36:44.801 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:36:44.801 INFO:teuthology.orchestra.run.smithi114.stdout:Install 30 Packages 2024-03-29T10:36:44.801 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 2 Packages 2024-03-29T10:36:44.801 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:36:44.801 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 58 M 2024-03-29T10:36:44.801 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:36:45.166 INFO:teuthology.orchestra.run.smithi176.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 783 kB/s | 548 kB 00:00 2024-03-29T10:36:45.283 INFO:teuthology.orchestra.run.smithi176.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-03-29T10:36:45.417 INFO:teuthology.orchestra.run.smithi176.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 231 kB/s | 31 kB 00:00 2024-03-29T10:36:45.475 INFO:teuthology.orchestra.run.smithi176.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.8 MB/s | 2.0 MB 00:00 2024-03-29T10:36:45.533 INFO:teuthology.orchestra.run.smithi176.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-29T10:36:45.750 INFO:teuthology.orchestra.run.smithi114.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 9.0 MB/s | 5.3 MB 00:00 2024-03-29T10:36:45.793 INFO:teuthology.orchestra.run.smithi114.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 605 kB/s | 25 kB 00:00 2024-03-29T10:36:45.876 INFO:teuthology.orchestra.run.smithi114.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 8.7 MB/s | 742 kB 00:00 2024-03-29T10:36:45.943 INFO:teuthology.orchestra.run.smithi114.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T10:36:46.259 INFO:teuthology.orchestra.run.smithi176.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 469 kB/s | 863 kB 00:01 2024-03-29T10:36:46.264 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:46.264 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.8 MB/s | 6.9 MB 00:02 2024-03-29T10:36:46.284 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:36:46.303 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:36:46.303 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:36:46.321 INFO:teuthology.orchestra.run.smithi114.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 14 MB/s | 5.1 MB 00:00 2024-03-29T10:36:46.369 INFO:teuthology.orchestra.run.smithi114.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 902 kB/s | 45 kB 00:00 2024-03-29T10:36:46.427 INFO:teuthology.orchestra.run.smithi114.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T10:36:46.474 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:36:46.474 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:36:46.477 INFO:teuthology.orchestra.run.smithi114.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.2 MB/s | 162 kB 00:00 2024-03-29T10:36:46.536 INFO:teuthology.orchestra.run.smithi114.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 5.4 MB/s | 320 kB 00:00 2024-03-29T10:36:46.646 INFO:teuthology.orchestra.run.smithi114.stdout:(10/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 6.6 MB/s | 9.7 MB 00:01 2024-03-29T10:36:46.670 INFO:teuthology.orchestra.run.smithi114.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.2 MB/s | 298 kB 00:00 2024-03-29T10:36:46.728 INFO:teuthology.orchestra.run.smithi114.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T10:36:46.795 INFO:teuthology.orchestra.run.smithi114.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2024-03-29T10:36:46.820 INFO:teuthology.orchestra.run.smithi114.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 703 kB/s | 106 kB 00:00 2024-03-29T10:36:46.846 INFO:teuthology.orchestra.run.smithi114.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-29T10:36:46.888 INFO:teuthology.orchestra.run.smithi114.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T10:36:46.913 INFO:teuthology.orchestra.run.smithi114.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-29T10:36:46.938 INFO:teuthology.orchestra.run.smithi114.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 41 kB 00:00 2024-03-29T10:36:46.964 INFO:teuthology.orchestra.run.smithi114.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T10:36:46.989 INFO:teuthology.orchestra.run.smithi114.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 698 kB/s | 35 kB 00:00 2024-03-29T10:36:47.013 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:36:47.014 INFO:teuthology.orchestra.run.smithi114.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 926 kB/s | 46 kB 00:00 2024-03-29T10:36:47.084 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:36:47.127 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:36:47.174 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:36:47.204 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:36:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:36:47.398 INFO:teuthology.orchestra.run.smithi114.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 756 kB/s | 309 kB 00:00 2024-03-29T10:36:47.431 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:36:47.465 INFO:teuthology.orchestra.run.smithi114.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-29T10:36:47.524 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:36:47.542 INFO:teuthology.orchestra.run.smithi114.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 631 kB/s | 49 kB 00:00 2024-03-29T10:36:47.567 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:36:47.596 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:36:47.609 INFO:teuthology.orchestra.run.smithi114.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-29T10:36:47.628 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:36:47.684 INFO:teuthology.orchestra.run.smithi114.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.6 MB/s | 4.4 MB 00:00 2024-03-29T10:36:47.751 INFO:teuthology.orchestra.run.smithi114.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-29T10:36:47.793 INFO:teuthology.orchestra.run.smithi114.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-29T10:36:47.868 INFO:teuthology.orchestra.run.smithi114.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-29T10:36:47.870 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:36:47.911 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:36:48.185 INFO:teuthology.orchestra.run.smithi114.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 8.6 MB/s | 3.4 MB 00:00 2024-03-29T10:36:48.452 INFO:teuthology.orchestra.run.smithi114.stdout:(31/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 6.0 MB/s | 20 MB 00:03 2024-03-29T10:36:48.636 INFO:teuthology.orchestra.run.smithi114.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 4.1 MB/s | 3.1 MB 00:00 2024-03-29T10:36:48.641 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:48.641 INFO:teuthology.orchestra.run.smithi114.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-29T10:36:48.641 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:36:48.676 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:36:48.676 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:36:48.717 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:36:48.717 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:36:48.717 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:36:48.717 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:36:48.717 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:36:48.717 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:36:48.717 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:36:48.717 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:36:48.717 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:36:48.717 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:36:48.718 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:36:48.718 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:36:49.076 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:36:49.077 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:36:49.078 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:36:49.078 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:49.078 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:36:49.146 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:36:49.146 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:36:49.337 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:36:49.900 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:36:50.052 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:36:50.053 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-29T10:36:50.053 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T10:36:50.053 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-29T10:36:50.053 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:36:50.053 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:36:50.053 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T10:36:50.053 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:36:50.053 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:36:50.053 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout:Install 12 Packages 2024-03-29T10:36:50.054 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:50.055 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 37 M 2024-03-29T10:36:50.055 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 209 M 2024-03-29T10:36:50.055 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:36:50.398 INFO:teuthology.orchestra.run.smithi176.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 195 kB/s | 32 kB 00:00 2024-03-29T10:36:50.423 INFO:teuthology.orchestra.run.smithi176.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 698 kB/s | 17 kB 00:00 2024-03-29T10:36:50.449 INFO:teuthology.orchestra.run.smithi176.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-03-29T10:36:50.516 INFO:teuthology.orchestra.run.smithi176.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:36:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:36:50.691 INFO:teuthology.orchestra.run.smithi176.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T10:36:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:36:50.758 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:36:50.758 INFO:teuthology.orchestra.run.smithi176.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-29T10:36:50.791 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:36:50.792 INFO:teuthology.orchestra.run.smithi176.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T10:36:50.822 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:36:50.839 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:36:50.997 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:36:51.053 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:36:51.092 INFO:teuthology.orchestra.run.smithi176.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-29T10:36:51.209 INFO:teuthology.orchestra.run.smithi176.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 7.6 MB/s | 7.4 MB 00:00 2024-03-29T10:36:51.318 INFO:teuthology.orchestra.run.smithi176.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-29T10:36:51.401 INFO:teuthology.orchestra.run.smithi176.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-29T10:36:51.435 INFO:teuthology.orchestra.run.smithi176.stdout:(12/12): libquadmath-11.4.1-3.el9.x86_64.rpm 571 kB/s | 195 kB 00:00 2024-03-29T10:36:51.437 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:51.437 INFO:teuthology.orchestra.run.smithi176.stdout:Total 27 MB/s | 37 MB 00:01 2024-03-29T10:36:51.452 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:36:51.478 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:36:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:36:51.502 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:36:51.502 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:36:51.569 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:36:51.666 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:36:51.757 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:36:51.786 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:36:51.830 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:36:51.869 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:36:51.869 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:36:51.899 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:36:51.922 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:36:51.968 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:36:52.004 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:36:52.114 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:36:52.146 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:36:52.187 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:36:52.220 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:36:52.248 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:36:52.281 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:36:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:36:52.596 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:36:52.628 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:36:52.657 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:36:52.689 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:36:52.709 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:36:52.741 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:36:52.782 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:36:52.798 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:36:52.850 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:36:53.026 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:36:53.058 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:36:53.134 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:36:53.245 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:36:53.280 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:36:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:36:53.419 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:36:53.449 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:36:53.795 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:36:53.902 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:36:55.137 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:36:55.619 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:36:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:36:56.167 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:36:56.198 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:36:56.229 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:36:56.274 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:36:56.363 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:36:56.363 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:36:56.363 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:36:56.363 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:36:56.392 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:36:58.788 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:36:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:36:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:59.234 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:36:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:36:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:36:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:36:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:36:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:36:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:36:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:36:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:36:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:36:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:36:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:36:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:36:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:36:59.235 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:36:59.666 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:37:00.240 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:21 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:00.390 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:00.391 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T10:37:00.391 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:37:00.392 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:37:00.393 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:37:00.393 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:37:00.393 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:00.393 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:37:00.393 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T10:37:00.393 INFO:teuthology.orchestra.run.smithi176.stdout:Install 13 Packages 2024-03-29T10:37:00.393 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:00.394 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 2.0 M 2024-03-29T10:37:00.394 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 26 M 2024-03-29T10:37:00.394 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:37:00.674 INFO:teuthology.orchestra.run.smithi176.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 200 kB/s | 24 kB 00:00 2024-03-29T10:37:00.699 INFO:teuthology.orchestra.run.smithi176.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-29T10:37:00.749 INFO:teuthology.orchestra.run.smithi176.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:37:00.774 INFO:teuthology.orchestra.run.smithi176.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:37:00.800 INFO:teuthology.orchestra.run.smithi176.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:37:00.825 INFO:teuthology.orchestra.run.smithi176.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-29T10:37:00.850 INFO:teuthology.orchestra.run.smithi176.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 175 kB/s | 52 kB 00:00 2024-03-29T10:37:00.942 INFO:teuthology.orchestra.run.smithi176.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-29T10:37:00.970 INFO:teuthology.orchestra.run.smithi176.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-29T10:37:01.059 INFO:teuthology.orchestra.run.smithi176.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 971 kB/s | 203 kB 00:00 2024-03-29T10:37:01.085 INFO:teuthology.orchestra.run.smithi176.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-03-29T10:37:01.118 INFO:teuthology.orchestra.run.smithi176.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-29T10:37:01.202 INFO:teuthology.orchestra.run.smithi176.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-29T10:37:01.203 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:01.204 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2024-03-29T10:37:01.204 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:37:01.217 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:37:01.217 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:37:01.307 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:37:01.307 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:37:01.820 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:37:01.990 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:37:02.044 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:37:02.099 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:37:02.129 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:37:02.169 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:37:02.263 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:37:02.372 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:37:02.430 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:37:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:37:03.193 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:37:03.228 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:37:03.275 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:37:03.297 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:37:04.093 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:37:04.093 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:37:04.093 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:37:04.093 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:37:04.093 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:37:04.093 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:37:04.093 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:37:04.093 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:37:04.093 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:37:04.093 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:37:04.094 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:37:04.094 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:37:04.094 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:37:04.447 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:37:04.448 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:37:04.449 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:37:04.449 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:37:04.449 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:04.449 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:04.707 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:37:05.274 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:05.426 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:05.427 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T10:37:05.427 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T10:37:05.427 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T10:37:05.427 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:37:05.427 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:37:05.427 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T10:37:05.427 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:37:05.427 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:37:05.427 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:37:05.428 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:37:05.428 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:37:05.428 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:05.428 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:37:05.428 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T10:37:05.428 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-03-29T10:37:05.428 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:05.429 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 6.8 M 2024-03-29T10:37:05.429 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 33 M 2024-03-29T10:37:05.429 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:37:05.978 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-29T10:37:06.042 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:37:06.043 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-03-29T10:37:06.043 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-03-29T10:37:06.043 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-03-29T10:37:06.043 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-03-29T10:37:06.043 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-03-29T10:37:06.043 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-03-29T10:37:06.043 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-03-29T10:37:06.043 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:06.053 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T10:37:06.078 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 454 kB/s | 138 kB 00:00 2024-03-29T10:37:06.362 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 187 kB/s | 58 kB 00:00 2024-03-29T10:37:06.387 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2024-03-29T10:37:06.487 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T10:37:06.489 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:06.489 INFO:teuthology.orchestra.run.smithi176.stdout:Total 6.4 MB/s | 6.8 MB 00:01 2024-03-29T10:37:06.503 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:37:06.513 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:37:06.513 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:37:06.601 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:37:06.602 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:37:06.899 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:37:06.957 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:37:06.982 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:37:06.982 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:06.982 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:37:06.982 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:37:06.982 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:37:06.982 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:06.984 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:37:07.054 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:37:07.054 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:37:07.077 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:37:07.132 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:37:07.639 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:37:07.735 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:37:07.803 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:37:07.826 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:37:08.194 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:37:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:37:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:37:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:37:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:37:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:37:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:37:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:37:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:37:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:37:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:37:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:37:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:37:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:37:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:37:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:37:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:37:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:37:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:37:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:37:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:37:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:37:08.198 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:37:08.642 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:37:08.643 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:37:08.643 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:37:08.643 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:37:08.643 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:37:08.643 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:37:08.946 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:37:08.946 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:08.947 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:37:08.947 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:08.947 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:37:08.947 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:37:08.947 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:37:08.947 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:37:08.947 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:37:08.947 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:08.947 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:09.205 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-fuse 2024-03-29T10:37:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:37:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:09.337 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-03-29T10:37:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.337 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:37:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:37:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:37:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:37:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:37:09.340 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:37:09.340 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:09.340 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:37:09.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-test 2024-03-29T10:37:09.767 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:09.916 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:09.917 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:09.917 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T10:37:09.917 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:09.917 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:37:09.917 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:37:09.918 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T10:37:09.918 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:37:09.918 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:09.918 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:37:09.919 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:09.919 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-03-29T10:37:09.919 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:09.919 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 964 k 2024-03-29T10:37:09.919 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 2.8 M 2024-03-29T10:37:09.919 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:37:10.047 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 608 kB/s | 79 kB 00:00 2024-03-29T10:37:10.198 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:37:10.230 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 885 kB 00:00 2024-03-29T10:37:10.231 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:10.232 INFO:teuthology.orchestra.run.smithi176.stdout:Total 3.0 MB/s | 964 kB 00:00 2024-03-29T10:37:10.232 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:37:10.253 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:37:10.253 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:37:10.326 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:37:10.327 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:37:10.356 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:37:10.357 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:37:10.357 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T10:37:10.357 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:37:10.357 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:37:10.357 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:37:10.357 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T10:37:10.358 INFO:teuthology.orchestra.run.smithi114.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:37:10.358 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:37:10.358 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:37:10.358 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:37:10.358 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:37:10.358 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:10.358 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:37:10.358 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:37:10.358 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-03-29T10:37:10.358 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:10.359 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 48 M 2024-03-29T10:37:10.359 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 204 M 2024-03-29T10:37:10.359 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:37:10.507 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:37:10.641 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:37:10.648 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:37:10.714 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:37:10.742 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.9 MB/s | 300 kB 00:00 2024-03-29T10:37:10.767 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 556 kB/s | 163 kB 00:00 2024-03-29T10:37:10.792 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T10:37:10.984 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 864 kB/s | 187 kB 00:00 2024-03-29T10:37:11.379 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:37:11.379 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:37:11.665 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:37:11.666 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:11.666 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:37:11.666 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:37:11.666 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:11.666 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:11.927 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-volume 2024-03-29T10:37:12.507 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:12.665 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:12.665 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:12.666 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T10:37:12.666 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:12.666 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:37:12.666 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:37:12.666 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T10:37:12.666 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:37:12.666 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:12.666 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:37:12.666 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:12.666 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-03-29T10:37:12.666 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:12.667 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 332 k 2024-03-29T10:37:12.667 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 1.5 M 2024-03-29T10:37:12.667 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:37:12.794 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 544 kB/s | 69 kB 00:00 2024-03-29T10:37:12.961 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 899 kB/s | 264 kB 00:00 2024-03-29T10:37:12.962 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:12.962 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-29T10:37:12.963 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:37:12.971 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:37:12.971 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:37:13.013 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:37:13.013 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:37:13.017 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 19 MB/s | 47 MB 00:02 2024-03-29T10:37:13.021 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:13.021 INFO:teuthology.orchestra.run.smithi114.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-29T10:37:13.035 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:37:13.056 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:37:13.057 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:37:13.162 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:37:13.353 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:37:13.388 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:37:13.404 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:37:13.404 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:13.404 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:37:13.404 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:13.435 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:37:13.435 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:37:14.001 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:37:14.076 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:37:14.116 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:37:14.147 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:37:14.190 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:37:14.193 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:37:14.221 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:37:14.497 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:37:14.497 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:14.498 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:37:14.498 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:14.498 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:37:14.498 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:14.498 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:14.799 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados-devel 2024-03-29T10:37:15.380 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:15.531 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:15.531 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:15.531 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-29T10:37:15.531 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:15.531 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:37:15.531 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:37:15.532 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:15.532 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:37:15.532 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:15.532 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T10:37:15.532 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:15.532 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 127 k 2024-03-29T10:37:15.532 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 456 k 2024-03-29T10:37:15.533 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:37:15.875 INFO:teuthology.orchestra.run.smithi176.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 370 kB/s | 127 kB 00:00 2024-03-29T10:37:15.875 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:15.876 INFO:teuthology.orchestra.run.smithi176.stdout:Total 368 kB/s | 127 kB 00:00 2024-03-29T10:37:15.876 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:37:15.881 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:37:15.881 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:37:15.937 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:37:15.938 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:37:16.089 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:37:16.182 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:16.645 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:16.938 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:16.939 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:16.939 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:37:16.939 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:16.939 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:16.939 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:17.195 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs2 2024-03-29T10:37:17.771 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:17.872 INFO:teuthology.orchestra.run.smithi176.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:17.924 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:17.925 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T10:37:17.926 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:17.999 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs-devel 2024-03-29T10:37:18.570 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:18.681 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:37:18.720 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:18.721 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:18.721 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-29T10:37:18.721 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:18.721 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:37:18.721 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:37:18.721 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:18.721 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:37:18.721 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:18.722 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T10:37:18.722 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:18.722 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 31 k 2024-03-29T10:37:18.722 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 140 k 2024-03-29T10:37:18.722 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:37:18.762 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:37:18.888 INFO:teuthology.orchestra.run.smithi176.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 188 kB/s | 31 kB 00:00 2024-03-29T10:37:18.888 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:18.888 INFO:teuthology.orchestra.run.smithi176.stdout:Total 187 kB/s | 31 kB 00:00 2024-03-29T10:37:18.888 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:37:18.892 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:37:18.892 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:37:18.926 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:37:18.926 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:37:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:37:19.154 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:19.548 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:19.793 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:37:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:19.793 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:19.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados2 2024-03-29T10:37:20.524 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:20.625 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:20.676 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:20.677 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T10:37:20.677 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:20.728 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librbd1 2024-03-29T10:37:21.216 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:37:21.216 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:37:21.217 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:37:21.217 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:37:21.217 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:37:21.217 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:37:21.290 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:21.387 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:21.438 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:21.439 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T10:37:21.439 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:21.486 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rados 2024-03-29T10:37:21.586 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:37:21.586 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:21.586 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:37:21.586 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:21.586 INFO:teuthology.orchestra.run.smithi114.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:37:21.586 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:21.587 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:37:21.587 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:37:21.587 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:37:21.587 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:21.587 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:37:21.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph 2024-03-29T10:37:22.050 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:22.148 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:22.200 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:22.201 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T10:37:22.201 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:22.257 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rgw 2024-03-29T10:37:22.501 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:37:22.675 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:37:22.678 INFO:teuthology.orchestra.run.smithi114.stdout:========================================================================================= 2024-03-29T10:37:22.678 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T10:37:22.678 INFO:teuthology.orchestra.run.smithi114.stdout:========================================================================================= 2024-03-29T10:37:22.678 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:37:22.678 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:37:22.678 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T10:37:22.678 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:37:22.679 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:37:22.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:22.681 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:37:22.682 INFO:teuthology.orchestra.run.smithi114.stdout:========================================================================================= 2024-03-29T10:37:22.682 INFO:teuthology.orchestra.run.smithi114.stdout:Install 36 Packages 2024-03-29T10:37:22.682 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:22.682 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 30 M 2024-03-29T10:37:22.683 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 107 M 2024-03-29T10:37:22.683 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:37:22.821 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:22.920 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:22.972 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:22.973 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T10:37:22.973 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:23.024 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-cephfs 2024-03-29T10:37:23.438 INFO:teuthology.orchestra.run.smithi114.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-29T10:37:23.592 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:23.690 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:23.742 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:23.743 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T10:37:23.743 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:23.794 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rbd 2024-03-29T10:37:23.955 INFO:teuthology.orchestra.run.smithi114.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.5 MB/s | 2.1 MB 00:00 2024-03-29T10:37:24.357 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:24.405 INFO:teuthology.orchestra.run.smithi114.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 1.5 MB 00:01 2024-03-29T10:37:24.454 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:24.506 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:24.507 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T10:37:24.507 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:24.555 INFO:teuthology.orchestra.run.smithi114.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.6 MB/s | 247 kB 00:00 2024-03-29T10:37:24.564 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-fuse 2024-03-29T10:37:24.681 INFO:teuthology.orchestra.run.smithi114.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T10:37:24.722 INFO:teuthology.orchestra.run.smithi114.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:37:24.748 INFO:teuthology.orchestra.run.smithi114.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T10:37:24.773 INFO:teuthology.orchestra.run.smithi114.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-29T10:37:24.798 INFO:teuthology.orchestra.run.smithi114.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-29T10:37:24.841 INFO:teuthology.orchestra.run.smithi114.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T10:37:24.907 INFO:teuthology.orchestra.run.smithi114.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T10:37:24.941 INFO:teuthology.orchestra.run.smithi114.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T10:37:24.974 INFO:teuthology.orchestra.run.smithi114.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T10:37:25.008 INFO:teuthology.orchestra.run.smithi114.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T10:37:25.130 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:25.192 INFO:teuthology.orchestra.run.smithi114.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 681 kB/s | 125 kB 00:00 2024-03-29T10:37:25.267 INFO:teuthology.orchestra.run.smithi114.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-03-29T10:37:25.279 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:25.280 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:25.280 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-29T10:37:25.280 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:25.280 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:37:25.280 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:37:25.280 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:25.280 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:37:25.280 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:25.280 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T10:37:25.280 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:25.281 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 85 k 2024-03-29T10:37:25.281 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 231 k 2024-03-29T10:37:25.281 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:37:25.559 INFO:teuthology.orchestra.run.smithi114.stdout:(17/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.2 MB/s | 4.7 MB 00:02 2024-03-29T10:37:25.651 INFO:teuthology.orchestra.run.smithi114.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-03-29T10:37:25.657 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 227 kB/s | 85 kB 00:00 2024-03-29T10:37:25.658 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:25.658 INFO:teuthology.orchestra.run.smithi176.stdout:Total 226 kB/s | 85 kB 00:00 2024-03-29T10:37:25.658 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:37:25.665 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:37:25.665 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:37:25.722 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:37:25.722 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:37:25.859 INFO:teuthology.orchestra.run.smithi114.stdout:(19/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 8.8 MB/s | 17 MB 00:01 2024-03-29T10:37:25.861 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:37:25.876 INFO:teuthology.orchestra.run.smithi114.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 137 kB/s | 43 kB 00:00 2024-03-29T10:37:25.901 INFO:teuthology.orchestra.run.smithi114.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 686 kB/s | 172 kB 00:00 2024-03-29T10:37:25.964 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:25.968 INFO:teuthology.orchestra.run.smithi114.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-29T10:37:25.985 INFO:teuthology.orchestra.run.smithi114.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-03-29T10:37:26.061 INFO:teuthology.orchestra.run.smithi114.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 312 kB/s | 23 kB 00:00 2024-03-29T10:37:26.086 INFO:teuthology.orchestra.run.smithi114.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 195 kB/s | 19 kB 00:00 2024-03-29T10:37:26.145 INFO:teuthology.orchestra.run.smithi114.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-03-29T10:37:26.203 INFO:teuthology.orchestra.run.smithi114.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 395 kB/s | 46 kB 00:00 2024-03-29T10:37:26.237 INFO:teuthology.orchestra.run.smithi114.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 857 kB/s | 79 kB 00:00 2024-03-29T10:37:26.287 INFO:teuthology.orchestra.run.smithi114.stdout:(29/36): python3-cherrypy-18.6.1-2.el9.noarch.r 839 kB/s | 358 kB 00:00 2024-03-29T10:37:26.321 INFO:teuthology.orchestra.run.smithi114.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-29T10:37:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-29T10:37:26.374 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:26.396 INFO:teuthology.orchestra.run.smithi114.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-29T10:37:26.422 INFO:teuthology.orchestra.run.smithi114.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-03-29T10:37:26.505 INFO:teuthology.orchestra.run.smithi114.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-03-29T10:37:26.531 INFO:teuthology.orchestra.run.smithi114.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-29T10:37:26.556 INFO:teuthology.orchestra.run.smithi114.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-03-29T10:37:26.561 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:26.561 INFO:teuthology.orchestra.run.smithi114.stdout:Total 7.6 MB/s | 30 MB 00:03 2024-03-29T10:37:26.587 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:37:26.630 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:37:26.630 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:37:26.692 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:26.692 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:26.693 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:37:26.693 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:26.693 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:26.693 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:26.938 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror 2024-03-29T10:37:27.041 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:37:27.042 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:37:27.526 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:27.684 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:27.685 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:27.685 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-29T10:37:27.685 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:27.685 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:37:27.686 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:37:27.686 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:27.686 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:37:27.686 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:27.686 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T10:37:27.686 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:27.686 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.1 M 2024-03-29T10:37:27.686 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 13 M 2024-03-29T10:37:27.686 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:37:28.144 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 6.8 MB/s | 3.1 MB 00:00 2024-03-29T10:37:28.144 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:28.144 INFO:teuthology.orchestra.run.smithi176.stdout:Total 6.7 MB/s | 3.1 MB 00:00 2024-03-29T10:37:28.145 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:37:28.154 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:37:28.154 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:37:28.228 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:37:28.228 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:37:28.392 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:37:28.467 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:37:28.508 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:37:28.536 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:37:28.605 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:37:28.643 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:37:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:37:28.675 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:37:28.711 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:28.736 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:28.736 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:28.736 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:37:28.736 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:37:28.736 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:37:28.737 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:28.766 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:37:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:37:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:37:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:37:29.012 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:37:29.050 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:37:29.099 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:37:29.155 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:37:29.337 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:37:29.388 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:37:29.418 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:37:29.453 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:37:29.485 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:37:29.561 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:37:29.596 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:37:29.629 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:37:29.669 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:37:29.704 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:37:29.738 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:37:29.827 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:37:29.869 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:37:29.877 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:29.878 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:29.878 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:37:29.878 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:29.878 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:29.878 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:29.941 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:37:30.078 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:37:30.097 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-nbd 2024-03-29T10:37:30.196 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:37:30.353 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:37:30.374 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:37:30.400 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:37:30.400 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:30.400 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:37:30.400 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:37:30.400 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:37:30.401 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:30.669 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 10:35:39 AM UTC. 2024-03-29T10:37:30.817 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T10:37:30.818 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:30.818 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-29T10:37:30.818 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:30.818 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T10:37:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:37:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:30.819 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T10:37:30.819 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T10:37:30.819 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T10:37:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:30.819 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 171 k 2024-03-29T10:37:30.819 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 494 k 2024-03-29T10:37:30.819 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T10:37:31.030 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 813 kB/s | 171 kB 00:00 2024-03-29T10:37:31.031 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:31.031 INFO:teuthology.orchestra.run.smithi176.stdout:Total 807 kB/s | 171 kB 00:00 2024-03-29T10:37:31.032 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T10:37:31.039 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T10:37:31.039 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T10:37:31.096 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T10:37:31.096 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T10:37:31.243 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T10:37:31.345 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:31.817 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:31.841 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:37:31.868 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:37:31.868 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:31.868 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:37:31.868 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:37:31.868 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:37:31.868 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:32.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:32.132 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:32.132 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T10:37:32.132 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:32.132 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:37:32.132 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T10:37:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:37:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:37:32.332 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:32.332 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:37:32.332 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:37:32.332 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:37:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:32.342 DEBUG:teuthology.parallel:result is None 2024-03-29T10:37:32.543 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:37:32.566 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:37:32.566 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:32.566 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:37:32.566 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:37:32.566 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:37:32.566 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:32.633 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:37:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:37:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:37:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:37:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:37:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:37:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:37:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:37:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:37:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:37:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:37:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:37:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:37:35.115 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:37:35.116 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:37:35.117 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:35.118 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:37:35.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-base 2024-03-29T10:37:36.039 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:37:36.139 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:36.189 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:37:36.190 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T10:37:36.190 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:37:36.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install cephadm 2024-03-29T10:37:36.828 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:37:36.978 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:37:36.979 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:37:36.979 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T10:37:36.980 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:37:36.980 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:37:36.980 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:37:36.980 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:36.980 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:37:36.980 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:37:36.981 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T10:37:36.981 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:36.981 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 747 k 2024-03-29T10:37:36.981 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 754 k 2024-03-29T10:37:36.981 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:37:37.275 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.5 MB/s | 747 kB 00:00 2024-03-29T10:37:37.275 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:37.276 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.5 MB/s | 747 kB 00:00 2024-03-29T10:37:37.276 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:37:37.281 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:37:37.281 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:37:37.287 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:37:37.287 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:37:37.363 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:37:37.801 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:37.896 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:38.307 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:38.590 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:38.590 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:38.590 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:37:38.590 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:38.591 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:38.591 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:37:38.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:37:39.375 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:37:39.525 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:37:39.526 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:37:39.526 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-29T10:37:39.526 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:37:39.526 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:37:39.526 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:37:39.527 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:39.527 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:37:39.527 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:37:39.527 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T10:37:39.527 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:39.527 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 145 k 2024-03-29T10:37:39.527 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 436 k 2024-03-29T10:37:39.527 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:37:40.148 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 234 kB/s | 145 kB 00:00 2024-03-29T10:37:40.148 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:40.149 INFO:teuthology.orchestra.run.smithi114.stdout:Total 234 kB/s | 145 kB 00:00 2024-03-29T10:37:40.149 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:37:40.157 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:37:40.157 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:37:40.211 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:37:40.212 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:37:40.355 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:37:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:37:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:37:40.431 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:40.431 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:37:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:41.393 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:37:41.393 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:41.393 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:37:41.393 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:41.393 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:41.393 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:37:41.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr 2024-03-29T10:37:42.226 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:37:42.324 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:42.378 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:37:42.379 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T10:37:42.379 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:37:42.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:37:43.022 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:37:43.177 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:37:43.177 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:37:43.178 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 6.9 M 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-03-29T10:37:43.179 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:37:43.559 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 163 kB/s | 31 kB 00:00 2024-03-29T10:37:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 51 kB/s | 15 kB 00:00 2024-03-29T10:37:43.735 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-29T10:37:43.760 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-29T10:37:43.802 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.1 MB/s | 1.8 MB 00:00 2024-03-29T10:37:43.877 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-29T10:37:43.969 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.6 MB/s | 548 kB 00:00 2024-03-29T10:37:44.019 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-03-29T10:37:44.053 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 919 kB/s | 31 kB 00:00 2024-03-29T10:37:44.103 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.0 MB/s | 2.0 MB 00:00 2024-03-29T10:37:44.137 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.1 MB/s | 863 kB 00:00 2024-03-29T10:37:44.162 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-03-29T10:37:44.164 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:44.164 INFO:teuthology.orchestra.run.smithi114.stdout:Total 6.9 MB/s | 6.9 MB 00:00 2024-03-29T10:37:44.182 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:37:44.199 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:37:44.199 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:37:44.370 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:37:44.370 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:37:44.886 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:37:44.954 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:37:44.997 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:37:45.044 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:37:45.092 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:37:45.202 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:37:45.323 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:37:45.410 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:37:45.454 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:37:45.491 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:37:45.522 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:37:45.762 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:37:45.789 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:37:46.520 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:37:46.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:37:46.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:37:46.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:37:46.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:37:46.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:37:46.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:37:46.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:37:46.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:37:46.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:37:46.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:37:46.521 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:37:46.868 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:37:46.868 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:46.868 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:37:46.868 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:37:46.868 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:46.868 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:46.869 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:46.869 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:37:46.869 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:37:46.869 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:37:46.869 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:37:46.869 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:37:46.869 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:37:46.869 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:37:46.869 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:37:46.869 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:46.869 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:37:47.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:37:47.708 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:37:47.862 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:37:47.863 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-29T10:37:47.863 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T10:37:47.863 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-29T10:37:47.863 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:37:47.863 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:37:47.863 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:47.864 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:37:47.865 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-29T10:37:47.865 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-03-29T10:37:47.865 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:47.865 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 37 M 2024-03-29T10:37:47.865 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 209 M 2024-03-29T10:37:47.865 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:37:48.176 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 240 kB/s | 32 kB 00:00 2024-03-29T10:37:48.218 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 424 kB/s | 17 kB 00:00 2024-03-29T10:37:48.260 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 34 kB 00:00 2024-03-29T10:37:48.328 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:37:48.528 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-29T10:37:48.586 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T10:37:48.611 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-29T10:37:48.728 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.9 MB/s | 803 kB 00:00 2024-03-29T10:37:48.904 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-03-29T10:37:48.946 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 910 kB/s | 195 kB 00:00 2024-03-29T10:37:49.296 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 5.9 MB/s | 7.4 MB 00:01 2024-03-29T10:37:49.362 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 535 kB/s | 245 kB 00:00 2024-03-29T10:37:49.366 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:49.366 INFO:teuthology.orchestra.run.smithi114.stdout:Total 25 MB/s | 37 MB 00:01 2024-03-29T10:37:49.406 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:37:49.430 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:37:49.431 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:37:49.796 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:37:49.796 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:37:50.495 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:37:50.561 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:37:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:37:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:37:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:37:50.897 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:37:50.927 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:37:51.007 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:37:51.644 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:37:51.764 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:37:53.035 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:37:54.057 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:37:54.104 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:37:56.694 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:37:56.694 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:37:56.694 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:37:56.695 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:37:56.695 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:37:56.695 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:37:56.695 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:37:56.695 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:37:56.695 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:37:56.695 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:37:56.695 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:37:56.695 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:37:57.162 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:37:57.162 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:57.162 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:37:57.162 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:57.162 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:37:57.162 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:37:57.162 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:37:57.162 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:37:57.163 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:37:57.163 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:37:57.163 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:37:57.163 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:37:57.163 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:37:57.163 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:37:57.163 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:37:57.163 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:57.163 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:37:57.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:37:58.114 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:16 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:37:58.266 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:37:58.267 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T10:37:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T10:37:58.267 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T10:37:58.267 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:37:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:37:58.267 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T10:37:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:37:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:37:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:37:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2024-03-29T10:37:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:37:58.269 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 2.0 M 2024-03-29T10:37:58.269 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-03-29T10:37:58.269 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:37:58.744 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 208 kB/s | 24 kB 00:00 2024-03-29T10:37:58.769 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2024-03-29T10:37:58.820 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:37:58.845 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:37:58.870 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:37:58.887 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T10:37:58.912 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 183 kB/s | 52 kB 00:00 2024-03-29T10:37:59.138 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-29T10:37:59.155 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-03-29T10:37:59.280 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-03-29T10:37:59.305 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 516 kB/s | 203 kB 00:00 2024-03-29T10:37:59.380 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 897 kB/s | 90 kB 00:00 2024-03-29T10:37:59.472 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-29T10:37:59.476 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:59.476 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-29T10:37:59.477 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:37:59.489 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:37:59.489 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:37:59.579 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:37:59.579 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:38:00.097 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:38:00.262 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:38:00.323 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:38:00.379 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:38:00.416 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:38:00.458 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:38:00.551 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:38:00.653 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:38:00.701 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:38:01.418 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:38:01.464 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:38:01.498 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:38:01.546 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:38:01.576 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:38:02.372 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:38:02.373 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:38:02.373 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:38:02.373 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:38:02.373 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:38:02.373 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:38:02.373 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:38:02.373 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:38:02.373 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:38:02.373 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:38:02.374 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:38:02.374 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:38:02.374 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:38:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:38:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:02.716 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:38:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:38:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:38:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:38:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:38:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:38:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:38:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:38:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:38:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:38:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:38:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:38:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:38:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:38:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:02.717 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:02.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:38:03.543 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:21 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:03.695 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:03.696 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:38:03.697 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T10:38:03.697 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-03-29T10:38:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:03.697 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 6.8 M 2024-03-29T10:38:03.697 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 33 M 2024-03-29T10:38:03.697 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:38:04.307 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T10:38:04.332 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 463 kB/s | 138 kB 00:00 2024-03-29T10:38:04.416 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 650 kB/s | 248 kB 00:00 2024-03-29T10:38:04.533 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 288 kB/s | 58 kB 00:00 2024-03-29T10:38:04.566 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 340 kB/s | 51 kB 00:00 2024-03-29T10:38:04.641 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T10:38:04.644 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:04.644 INFO:teuthology.orchestra.run.smithi114.stdout:Total 7.2 MB/s | 6.8 MB 00:00 2024-03-29T10:38:04.657 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:38:04.667 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:38:04.667 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:38:04.756 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:38:04.757 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:38:05.069 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:38:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:38:05.284 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:38:05.810 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:38:05.895 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:38:05.957 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:38:05.980 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:38:06.757 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:38:06.757 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:38:06.757 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:38:06.757 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:38:06.757 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:38:06.758 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:38:07.048 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:38:07.048 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:07.048 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:38:07.049 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:38:07.049 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:38:07.049 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:38:07.049 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:38:07.049 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:38:07.049 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:38:07.049 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:07.049 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:07.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-fuse 2024-03-29T10:38:07.858 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:08.008 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 964 k 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 2.8 M 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:38:08.143 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 590 kB/s | 79 kB 00:00 2024-03-29T10:38:08.818 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 885 kB 00:00 2024-03-29T10:38:08.820 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:08.820 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.2 MB/s | 964 kB 00:00 2024-03-29T10:38:08.820 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:38:08.841 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:38:08.841 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:38:08.918 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:38:08.918 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:38:09.110 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:38:09.236 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:38:09.319 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:38:09.973 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:38:09.973 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:38:10.307 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:38:10.307 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:10.307 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:38:10.307 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:38:10.307 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:10.307 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:10.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-volume 2024-03-29T10:38:11.162 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:11.315 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:11.316 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:11.316 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 332 k 2024-03-29T10:38:11.317 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 1.5 M 2024-03-29T10:38:11.318 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:38:11.440 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 563 kB/s | 69 kB 00:00 2024-03-29T10:38:11.881 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 468 kB/s | 264 kB 00:00 2024-03-29T10:38:11.882 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:11.883 INFO:teuthology.orchestra.run.smithi114.stdout:Total 587 kB/s | 332 kB 00:00 2024-03-29T10:38:11.883 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:38:11.892 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:38:11.892 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:38:11.932 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:38:11.932 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:38:12.076 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:38:12.272 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:38:12.317 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:38:12.333 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:38:12.333 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:12.334 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:38:12.334 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:13.039 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:38:13.366 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:38:13.367 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:13.367 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:38:13.367 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:38:13.367 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:38:13.367 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:13.367 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:13.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados-devel 2024-03-29T10:38:14.285 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:14.435 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:14.436 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:14.436 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-29T10:38:14.436 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:14.436 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:38:14.437 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:38:14.437 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:14.437 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:38:14.437 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:14.437 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T10:38:14.437 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:14.437 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 127 k 2024-03-29T10:38:14.437 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 456 k 2024-03-29T10:38:14.437 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:38:14.754 INFO:teuthology.orchestra.run.smithi114.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 400 kB/s | 127 kB 00:00 2024-03-29T10:38:14.754 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:14.755 INFO:teuthology.orchestra.run.smithi114.stdout:Total 398 kB/s | 127 kB 00:00 2024-03-29T10:38:14.755 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:38:14.761 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:38:14.761 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:38:14.816 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:38:14.816 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:38:14.955 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:38:15.069 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:15.524 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:15.798 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:15.798 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:15.798 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:38:15.798 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:15.799 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:15.799 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:16.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs2 2024-03-29T10:38:16.673 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:16.774 INFO:teuthology.orchestra.run.smithi114.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:16.827 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:16.828 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T10:38:16.828 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:16.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs-devel 2024-03-29T10:38:17.475 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:17.626 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:17.627 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-29T10:38:17.627 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:17.627 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:38:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:38:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:17.628 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:38:17.628 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:17.628 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T10:38:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:17.628 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 31 k 2024-03-29T10:38:17.629 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 140 k 2024-03-29T10:38:17.629 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:38:17.799 INFO:teuthology.orchestra.run.smithi114.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 182 kB/s | 31 kB 00:00 2024-03-29T10:38:17.800 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:17.800 INFO:teuthology.orchestra.run.smithi114.stdout:Total 181 kB/s | 31 kB 00:00 2024-03-29T10:38:17.800 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:38:17.804 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:38:17.804 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:38:17.833 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:38:17.833 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:38:17.965 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:38:18.090 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:18.764 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:18.764 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:18.764 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:38:18.764 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:18.764 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:18.764 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:18.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados2 2024-03-29T10:38:19.490 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:19.637 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:19.638 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T10:38:19.638 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:19.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librbd1 2024-03-29T10:38:20.255 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:20.352 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:20.404 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:20.405 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T10:38:20.405 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:20.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rados 2024-03-29T10:38:21.019 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:21.120 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:21.172 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:21.173 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T10:38:21.173 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:21.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rgw 2024-03-29T10:38:21.782 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:21.880 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:21.932 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:21.933 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T10:38:21.933 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:21.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-cephfs 2024-03-29T10:38:22.547 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:22.646 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:22.697 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:22.698 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T10:38:22.698 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:22.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rbd 2024-03-29T10:38:23.314 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:23.412 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:23.466 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:23.466 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T10:38:23.467 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:23.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-fuse 2024-03-29T10:38:24.080 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:24.228 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:24.228 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:24.228 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-03-29T10:38:24.228 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:24.229 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:38:24.229 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:38:24.229 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:24.229 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:38:24.229 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:24.229 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T10:38:24.229 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:24.229 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 85 k 2024-03-29T10:38:24.229 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 231 k 2024-03-29T10:38:24.229 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:38:24.511 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 303 kB/s | 85 kB 00:00 2024-03-29T10:38:24.511 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:24.511 INFO:teuthology.orchestra.run.smithi114.stdout:Total 301 kB/s | 85 kB 00:00 2024-03-29T10:38:24.511 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:38:24.518 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:38:24.518 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:38:24.572 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:38:24.573 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:38:24.706 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:38:24.800 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:25.200 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:25.490 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:25.490 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:25.490 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:38:25.490 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:25.490 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:25.491 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:25.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-mirror 2024-03-29T10:38:26.353 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:26.510 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:26.511 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:26.511 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-29T10:38:26.511 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:26.511 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:38:26.511 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:38:26.511 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:26.511 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:38:26.511 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:26.511 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T10:38:26.511 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:26.511 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.1 M 2024-03-29T10:38:26.512 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 13 M 2024-03-29T10:38:26.512 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.7 MB/s | 3.1 MB 00:01 2024-03-29T10:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.7 MB/s | 3.1 MB 00:01 2024-03-29T10:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:38:27.658 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:38:27.658 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:38:27.733 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:38:27.733 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:38:28.166 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:38:28.232 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:28.257 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:28.257 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:28.257 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:38:28.257 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:38:28.257 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:38:28.257 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:29.432 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:29.432 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:29.432 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:38:29.432 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:29.432 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:29.432 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:29.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-nbd 2024-03-29T10:38:30.292 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:36:42 AM UTC. 2024-03-29T10:38:30.442 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 171 k 2024-03-29T10:38:30.443 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 494 k 2024-03-29T10:38:30.444 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T10:38:30.803 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 478 kB/s | 171 kB 00:00 2024-03-29T10:38:30.803 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:30.803 INFO:teuthology.orchestra.run.smithi114.stdout:Total 476 kB/s | 171 kB 00:00 2024-03-29T10:38:30.803 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T10:38:30.811 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T10:38:30.811 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T10:38:30.866 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T10:38:30.866 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T10:38:31.012 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T10:38:31.109 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:31.552 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:31.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:31.836 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:31.836 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T10:38:31.836 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:31.836 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:31.836 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T10:38:32.089 DEBUG:teuthology.parallel:result is None 2024-03-29T10:38:32.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:38:32.234 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:38:32.254 INFO:teuthology.orchestra.run.smithi114.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:32.255 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:32.255 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:38:32.257 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:38:32.396 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:38:32.421 INFO:teuthology.orchestra.run.smithi176.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:32.421 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:32.422 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:38:32.423 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:38:32.424 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:38:32.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:38:32.452 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:38:32.452 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:38:32.494 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:38:32.494 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:38:32.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:38:32.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:38:32.588 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:38:32.588 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:38:32.619 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:38:32.689 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:38:32.690 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:38:32.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:38:32.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:38:32.782 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:38:32.782 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:38:32.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:38:32.882 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:38:32.882 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:38:32.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:38:32.909 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:38:32.975 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:38:32.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:38:33.002 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:38:33.070 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:38:33.153 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:38:33.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /var/log/ceph 2024-03-29T10:38:33.156 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /var/log/ceph 2024-03-29T10:38:33.184 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:38:33.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:38:33.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:38:33.250 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:38:33.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:38:33.288 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:38:33.327 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:38:33.327 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:38:33.327 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622645', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi114.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDR2+Wh4EFdavoNAlwXZOvp3kKLwaLJcfAlicDL6P656byf9my2aWpeW5A3gmSLnaGRIoRC2nm50uiRzDUcvhsI=', 'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApCkfzolq0sEtzqzhFSLQvasiZ2FY7OxW3B1Ce2+TbmTeFff2PlE1cXJlwWU7g+i40pHhoavGZC0srYa6M9OlY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:38:33.327 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:38:33.356 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:38:33.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:38:33.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:38:33.439 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:38:33.439 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:38:33.490 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:38:33.490 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-03-29T10:38:33.544 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:38:33.545 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:33.545 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T10:38:33.545 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:33.545 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:33.545 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-29 10:38:28.501702218 +0000 2024-03-29T10:38:33.545 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-29 10:34:49.582933742 +0000 2024-03-29T10:38:33.545 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-29 10:34:49.582933742 +0000 2024-03-29T10:38:33.545 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-29 10:34:49.582933742 +0000 2024-03-29T10:38:33.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:38:33.610 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-29T10:38:33.610 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-29T10:38:33.610 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000238958 s, 2.1 MB/s 2024-03-29T10:38:33.611 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:38:33.667 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-03-29T10:38:33.722 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:38:33.722 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:33.722 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T10:38:33.722 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:33.722 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:33.722 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-29 10:38:28.502702197 +0000 2024-03-29T10:38:33.722 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-29 10:34:49.920926499 +0000 2024-03-29T10:38:33.722 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-29 10:34:49.920926499 +0000 2024-03-29T10:38:33.722 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-29 10:34:49.920926499 +0000 2024-03-29T10:38:33.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:38:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-29T10:38:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-29T10:38:33.788 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000193778 s, 2.6 MB/s 2024-03-29T10:38:33.789 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:38:33.844 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-03-29T10:38:33.899 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:38:33.899 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:33.899 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T10:38:33.899 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:33.899 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:33.899 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-29 10:38:28.502702197 +0000 2024-03-29T10:38:33.899 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-29 10:34:50.266919084 +0000 2024-03-29T10:38:33.899 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-29 10:34:50.266919084 +0000 2024-03-29T10:38:33.900 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-29 10:34:50.266919084 +0000 2024-03-29T10:38:33.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:38:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-29T10:38:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-29T10:38:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000174291 s, 2.9 MB/s 2024-03-29T10:38:33.966 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:38:34.021 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-03-29T10:38:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:38:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:34.076 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T10:38:34.076 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:34.076 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:34.076 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-29 10:38:28.502702197 +0000 2024-03-29T10:38:34.076 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-29 10:34:50.612911670 +0000 2024-03-29T10:38:34.076 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-29 10:34:50.612911670 +0000 2024-03-29T10:38:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-29 10:34:50.612911670 +0000 2024-03-29T10:38:34.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:38:34.142 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-29T10:38:34.142 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-29T10:38:34.142 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000216762 s, 2.4 MB/s 2024-03-29T10:38:34.143 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:38:34.198 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:34.199 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:38:34.199 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:38:34.214 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:38:34.215 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-03-29T10:38:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:38:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:34.270 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T10:38:34.270 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:34.270 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:34.270 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-29 10:37:28.987445992 +0000 2024-03-29T10:38:34.270 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-29 10:34:49.580676764 +0000 2024-03-29T10:38:34.270 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-29 10:34:49.580676764 +0000 2024-03-29T10:38:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-29 10:34:49.580676764 +0000 2024-03-29T10:38:34.271 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:38:34.335 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-29T10:38:34.336 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-29T10:38:34.336 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000206687 s, 2.5 MB/s 2024-03-29T10:38:34.337 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:38:34.393 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-03-29T10:38:34.447 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:38:34.447 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:34.447 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T10:38:34.447 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:34.448 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:34.448 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-29 10:37:28.988445983 +0000 2024-03-29T10:38:34.448 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-29 10:34:50.101672140 +0000 2024-03-29T10:38:34.448 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-29 10:34:50.101672140 +0000 2024-03-29T10:38:34.448 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-29 10:34:50.101672140 +0000 2024-03-29T10:38:34.448 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:38:34.513 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-29T10:38:34.513 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-29T10:38:34.513 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000189756 s, 2.7 MB/s 2024-03-29T10:38:34.514 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:38:34.570 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-03-29T10:38:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:38:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:34.625 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T10:38:34.625 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:34.626 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:34.626 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-29 10:37:28.988445983 +0000 2024-03-29T10:38:34.626 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-29 10:34:50.433669193 +0000 2024-03-29T10:38:34.626 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-29 10:34:50.433669193 +0000 2024-03-29T10:38:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-29 10:34:50.433669193 +0000 2024-03-29T10:38:34.626 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:38:34.691 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-29T10:38:34.691 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-29T10:38:34.691 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000190771 s, 2.7 MB/s 2024-03-29T10:38:34.692 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:38:34.747 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-03-29T10:38:34.802 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:38:34.802 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:34.803 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T10:38:34.803 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:34.803 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:34.803 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-29 10:37:28.988445983 +0000 2024-03-29T10:38:34.803 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-29 10:34:50.766666238 +0000 2024-03-29T10:38:34.803 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-29 10:34:50.766666238 +0000 2024-03-29T10:38:34.803 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-29 10:34:50.766666238 +0000 2024-03-29T10:38:34.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:38:34.868 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-29T10:38:34.868 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-29T10:38:34.868 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000225276 s, 2.3 MB/s 2024-03-29T10:38:34.869 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:38:34.925 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:34.925 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:38:34.925 INFO:tasks.ceph:Generating config... 2024-03-29T10:38:34.926 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:38:34.927 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:38:34.927 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:38:34.927 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:38:34.927 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:38:34.927 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:38:34.927 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:38:34.927 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:38:34.927 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:38:34.927 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:38:34.927 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:38:34.928 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:38:34.928 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:38:34.928 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:38:34.928 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:38:34.928 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:38:34.928 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:38:34.928 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:38:34.928 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:38:34.928 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:38:34.928 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:38:34.928 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:38:34.928 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:38:34.928 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:38:34.928 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:38:34.929 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:38:34.929 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:38:34.929 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:38:34.929 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:38:34.929 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:38:34.929 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:38:34.929 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:38:34.929 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:38:34.929 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:38:34.929 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:38:34.929 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:38:34.930 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:38:34.931 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:38:34.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:38:34.991 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:38:34.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:38:35.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:38:35.105 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.114:6789]'), ('mon.c', '[v1:172.21.15.114:6790]'), ('mon.b', '[v1:172.21.15.176:6789]')] 2024-03-29T10:38:35.105 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.114:6789],[v1:172.21.15.114:6790],[v1:172.21.15.176:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:38:35.106 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:38:35.106 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:38:35.160 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.114:6789]' --addv c '[v1:172.21.15.114:6790]' --addv b '[v1:172.21.15.176:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:35.246 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:35.246 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: generated fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:38:35.246 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = quincy 2024-03-29T10:38:35.246 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 0 2024-03-29T10:38:35.246 INFO:teuthology.orchestra.run.smithi114.stdout:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:38:35.246 INFO:teuthology.orchestra.run.smithi114.stdout:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:38:35.246 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:38:35.246 INFO:teuthology.orchestra.run.smithi114.stdout:min_mon_release 17 (quincy) 2024-03-29T10:38:35.246 INFO:teuthology.orchestra.run.smithi114.stdout:election_strategy: 1 2024-03-29T10:38:35.247 INFO:teuthology.orchestra.run.smithi114.stdout:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:38:35.247 INFO:teuthology.orchestra.run.smithi114.stdout:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:38:35.247 INFO:teuthology.orchestra.run.smithi114.stdout:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:38:35.247 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:38:35.248 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:38:35.263 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID aa12b535-44a2-4ddb-9a33-48fb3a93e012... 2024-03-29T10:38:35.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:38:35.305 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:38:35.353 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-03-29T10:38:35.353 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-03-29T10:38:35.353 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:38:35.353 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-03-29T10:38:35.353 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.353 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-03-29T10:38:35.353 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.353 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-03-29T10:38:35.353 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.353 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.354 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.355 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "[v1:172.21.15.114:6789],[v1:172.21.15.114:6790],[v1:172.21.15.176:6789]" 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: lockdep = True 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: mon client directed command retry = 5 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: mon election default strategy = 1 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: ms bind msgr2 = False 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: ms inject socket failures = 5000 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: ms type = async 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.356 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-03-29T10:38:35.357 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.358 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-03-29T10:38:35.359 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-03-29T10:38:35.359 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-03-29T10:38:35.359 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore compression algorithm = zstd 2024-03-29T10:38:35.359 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore compression mode = aggressive 2024-03-29T10:38:35.359 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-03-29T10:38:35.359 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore zero block detection = True 2024-03-29T10:38:35.359 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 20 2024-03-29T10:38:35.359 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 20 2024-03-29T10:38:35.359 INFO:teuthology.orchestra.run.smithi114.stdout: debug monc = 20 2024-03-29T10:38:35.359 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-29T10:38:35.359 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-03-29T10:38:35.360 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 10 2024-03-29T10:38:35.360 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:38:35.360 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-03-29T10:38:35.360 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:38:35.360 INFO:teuthology.orchestra.run.smithi114.stdout: osd class default list = * 2024-03-29T10:38:35.360 INFO:teuthology.orchestra.run.smithi114.stdout: osd class load list = * 2024-03-29T10:38:35.360 INFO:teuthology.orchestra.run.smithi114.stdout: osd client watch timeout = 120 2024-03-29T10:38:35.360 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify cached snaps = True 2024-03-29T10:38:35.360 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify missing on start = True 2024-03-29T10:38:35.360 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock override recovery settings = True 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: osd op queue = debug_random 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: osd op queue cut off = debug_random 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: osd shutdown pgref assert = True 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:35.361 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-03-29T10:38:35.362 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-03-29T10:38:35.363 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:35.363 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.363 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:38:35.363 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:38:35.363 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:38:35.363 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.363 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:38:35.363 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:38:35.363 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:38:35.363 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: mon min osdmap epochs = 50 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune interval = 2 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune min = 15 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: mon scrub interval = 300 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on pool no app = False 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: paxos service trim min = 10 2024-03-29T10:38:35.364 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout: debug monc = 20 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout: debug objecter = 20 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout: debug rados = 20 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-03-29T10:38:35.365 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-03-29T10:38:35.366 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2024-03-29T10:38:35.366 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2024-03-29T10:38:35.366 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:38:35.366 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2024-03-29T10:38:35.366 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.366 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2024-03-29T10:38:35.367 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:38:35.368 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = "[v1:172.21.15.114:6789],[v1:172.21.15.114:6790],[v1:172.21.15.176:6789]" 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: lockdep = True 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: mon client directed command retry = 5 2024-03-29T10:38:35.369 INFO:teuthology.orchestra.run.smithi176.stdout: mon election default strategy = 1 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: ms bind msgr2 = False 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject socket failures = 5000 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: ms type = async 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2024-03-29T10:38:35.370 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug trim objects = true 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2024-03-29T10:38:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore compression algorithm = zstd 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore compression mode = aggressive 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore zero block detection = True 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 20 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 20 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: debug monc = 20 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2024-03-29T10:38:35.372 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 10 2024-03-29T10:38:35.373 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:38:35.373 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2024-03-29T10:38:35.373 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:38:35.373 INFO:teuthology.orchestra.run.smithi176.stdout: osd class default list = * 2024-03-29T10:38:35.373 INFO:teuthology.orchestra.run.smithi176.stdout: osd class load list = * 2024-03-29T10:38:35.373 INFO:teuthology.orchestra.run.smithi176.stdout: osd client watch timeout = 120 2024-03-29T10:38:35.373 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify cached snaps = True 2024-03-29T10:38:35.373 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify missing on start = True 2024-03-29T10:38:35.373 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:38:35.373 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock override recovery settings = True 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: osd op queue = debug_random 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: osd op queue cut off = debug_random 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: osd shutdown pgref assert = True 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2024-03-29T10:38:35.374 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:38:35.375 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: # 1m isn't quite enough 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: mon min osdmap epochs = 50 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune interval = 2 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune min = 15 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: mon scrub interval = 300 2024-03-29T10:38:35.376 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on pool no app = False 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: paxos service trim min = 10 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: debug monc = 20 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: debug objecter = 20 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: debug rados = 20 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.c] 2024-03-29T10:38:35.377 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.b] 2024-03-29T10:38:35.379 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:38:35.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:38:35.473 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:38:35.473 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:38:35.473 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:38:35.488 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:38:35.488 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:38:35.542 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T10:38:35.542 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:38:35.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:38:35.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:38:35.621 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:38:35.621 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:35.676 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T10:38:35.676 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:38:35.676 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:38:35.676 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:38:35.716 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:38:35.716 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:35.770 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:38:35.771 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:38:35.771 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:38:35.825 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:38:35.828 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:38:35.889 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:38:35.891 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:38:35.891 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:38:35.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:38:35.933 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:38:35.948 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:38:35.991 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:38:36.006 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:38:36.006 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:38:36.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:38:36.033 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:36.033 INFO:tasks.ceph:role: osd.0 2024-03-29T10:38:36.033 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T10:38:36.033 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:38:36.107 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:36.107 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:36.107 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:36.108 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:36.108 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:36.108 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:36.108 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:36.618 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-29T10:38:36.620 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-29T10:38:36.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:38:36.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:38:36.778 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:38:36.846 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:36.846 INFO:tasks.ceph:role: osd.1 2024-03-29T10:38:36.846 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T10:38:36.846 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:38:36.921 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:36.922 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:36.922 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:36.922 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:36.922 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:36.922 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:36.922 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:36.922 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:36.922 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:36.922 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:37.369 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-29T10:38:37.371 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-29T10:38:37.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:38:37.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:38:37.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:38:37.555 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:37.556 INFO:tasks.ceph:role: osd.2 2024-03-29T10:38:37.556 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T10:38:37.556 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:38:37.630 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:37.630 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:37.630 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:37.630 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:37.631 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:37.631 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:37.631 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:37.631 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:37.631 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:37.631 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:38.127 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-29T10:38:38.129 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-29T10:38:38.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:38:38.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:38:38.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:38:38.346 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:38.346 INFO:tasks.ceph:role: osd.3 2024-03-29T10:38:38.346 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T10:38:38.346 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:38:38.421 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:38.421 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:38.421 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:38.421 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:38.421 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:38.421 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:38.422 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:38.422 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:38.422 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:38.422 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:38.949 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-29T10:38:38.951 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-29T10:38:38.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:38:39.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:38:39.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:39.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:39.277+0000 7fc44fbfa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:38:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:39.278+0000 7fc44fbfa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:38:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:39.278+0000 7fc44fbfa640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:38:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:39.278+0000 7fc44fbfa640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:38:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:39.278+0000 7fc44fbfa640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:38:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:39.278+0000 7fc44fbfa640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:38:42.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:38:42.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:42.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:42.256+0000 7f1ebc019640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:38:42.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:42.256+0000 7f1ebc019640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:38:42.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:42.256+0000 7f1ebc019640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:38:42.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:42.256+0000 7f1ebc019640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:38:42.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:42.256+0000 7f1ebc019640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:38:42.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:42.257+0000 7f1ebc019640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:38:45.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:38:45.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:45.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:45.223+0000 7f378f411640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:38:45.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:45.223+0000 7f378f411640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:38:45.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:45.223+0000 7f378f411640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:38:45.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:45.223+0000 7f378f411640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:38:45.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:45.223+0000 7f378f411640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:38:45.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:45.223+0000 7f378f411640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:38:48.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:38:48.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:48.193+0000 7f7185224640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:38:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:48.193+0000 7f7185224640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:38:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:48.193+0000 7f7185224640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:38:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:48.193+0000 7f7185224640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:38:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:48.193+0000 7f7185224640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:38:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:38:48.194+0000 7f7185224640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:38:51.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:38:51.053 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:38:51.085 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:51.086 INFO:tasks.ceph:role: osd.4 2024-03-29T10:38:51.086 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T10:38:51.086 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:38:51.163 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:51.163 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:51.163 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:51.163 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:51.163 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:51.163 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:51.163 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:51.163 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:51.164 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:51.164 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:51.682 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-29T10:38:51.684 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-29T10:38:51.685 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:38:51.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:38:51.879 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:38:51.949 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:51.949 INFO:tasks.ceph:role: osd.5 2024-03-29T10:38:51.949 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T10:38:51.949 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:38:52.025 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:52.025 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:52.025 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:52.025 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:52.025 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:52.025 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:52.025 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:52.025 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:52.025 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:52.026 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:52.482 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-29T10:38:52.486 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-29T10:38:52.486 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:38:52.586 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:38:52.621 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:38:52.689 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:52.690 INFO:tasks.ceph:role: osd.6 2024-03-29T10:38:52.690 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T10:38:52.690 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:38:52.766 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:52.766 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:52.766 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:52.766 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:52.766 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:52.766 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:52.766 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:52.766 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:52.766 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:52.767 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:53.276 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-29T10:38:53.278 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-29T10:38:53.278 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:38:53.439 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:38:53.476 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:38:53.545 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:53.545 INFO:tasks.ceph:role: osd.7 2024-03-29T10:38:53.545 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T10:38:53.545 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:38:53.622 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:53.622 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:53.623 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:53.623 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:53.623 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:54.137 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-29T10:38:54.139 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-29T10:38:54.139 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:38:54.248 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:38:54.287 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:54.393 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:54.408+0000 7f78e0a22640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:38:54.393 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:54.408+0000 7f78e0a22640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:38:54.393 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:54.408+0000 7f78e0a22640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:38:54.393 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:54.408+0000 7f78e0a22640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:38:54.394 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:54.408+0000 7f78e0a22640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:38:54.394 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:54.409+0000 7f78e0a22640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:38:57.247 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:38:57.277 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:57.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:57.382+0000 7f275d448640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:38:57.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:57.382+0000 7f275d448640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:38:57.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:57.382+0000 7f275d448640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:38:57.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:57.382+0000 7f275d448640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:38:57.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:57.382+0000 7f275d448640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:38:57.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:38:57.383+0000 7f275d448640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:39:00.215 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:39:00.752 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:00.844 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:00.859+0000 7f6cb71aa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:39:00.844 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:00.859+0000 7f6cb71aa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:39:00.844 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:00.859+0000 7f6cb71aa640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:39:00.844 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:00.859+0000 7f6cb71aa640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:39:00.844 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:00.859+0000 7f6cb71aa640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:39:00.844 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:00.860+0000 7f6cb71aa640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:39:03.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:39:03.731 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:03.826 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:03.841+0000 7f5732087640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:39:03.826 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:03.841+0000 7f5732087640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:39:03.826 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:03.841+0000 7f5732087640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:39:03.826 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:03.841+0000 7f5732087640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:39:03.826 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:03.841+0000 7f5732087640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:39:03.826 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:03.842+0000 7f5732087640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:39:06.667 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:39:06.697 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:39:06.698 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:39:06.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:39:06.730 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:39:06.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:39:06.800 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:39:06.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:39:06.869 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:39:06.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:39:06.937 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:39:06.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:39:07.006 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:39:07.006 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:39:07.038 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:39:07.039 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:39:07.108 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:39:07.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:39:07.178 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:39:07.178 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:39:07.246 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:39:07.246 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:39:07.314 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:39:07.314 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:39:07.330 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:39:07.330 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:39:07.370 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:39:07.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:39:07.372 DEBUG:teuthology.orchestra.run.smithi176:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:39:07.413 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.y] 2024-03-29T10:39:07.413 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQArmgZmPO85MhAAa8ESkfqdy+HHw3CqQPbWrg== 2024-03-29T10:39:07.413 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.0] 2024-03-29T10:39:07.413 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAvmgZmKtGkEBAAQYMVvKU3R1bfG/ziER7W3Q== 2024-03-29T10:39:07.413 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.1] 2024-03-29T10:39:07.413 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAymgZmGQBeDxAAttw0F9vz/PxKStd5CVd1vg== 2024-03-29T10:39:07.413 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.2] 2024-03-29T10:39:07.413 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA1mgZmfFJeDRAA0IydSL609AcSLIiD0H7TYQ== 2024-03-29T10:39:07.413 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.3] 2024-03-29T10:39:07.413 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA4mgZmM5iZCxAAVOV0AClvBPK9qFZ+LSf3Kg== 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.x] 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQArmgZmJ839NRAAJA061nQ2v6/w7V9gV7sUag== 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.4] 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA+mgZmL/xjGBAAYD/zSdP2ETLpONUpZcA1Kw== 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.5] 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBBmgZmUAndFhAA5aKa0/PiCMr1+HmPsWuZ9g== 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.6] 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBEmgZmxaFLMxAADDwh2MpfL9GK5I4lJ8eLwg== 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.7] 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBHmgZmXtQ3MhAAv+yny/dPVag0m0xQINPRRg== 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-29T10:39:07.414 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQArmgZmUxGoOBAAs4QZnHcA7lkCfYmoNQiQvQ== 2024-03-29T10:39:07.415 INFO:teuthology.orchestra.run.smithi114.stdout:[client.1] 2024-03-29T10:39:07.415 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAsmgZmWbp5ABAA4xvyNc4usdjn7hAm5SPQeQ== 2024-03-29T10:39:07.435 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.y] 2024-03-29T10:39:07.435 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQArmgZmPO85MhAAa8ESkfqdy+HHw3CqQPbWrg== 2024-03-29T10:39:07.435 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.0] 2024-03-29T10:39:07.435 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAvmgZmKtGkEBAAQYMVvKU3R1bfG/ziER7W3Q== 2024-03-29T10:39:07.435 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.1] 2024-03-29T10:39:07.436 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAymgZmGQBeDxAAttw0F9vz/PxKStd5CVd1vg== 2024-03-29T10:39:07.436 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.2] 2024-03-29T10:39:07.436 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA1mgZmfFJeDRAA0IydSL609AcSLIiD0H7TYQ== 2024-03-29T10:39:07.436 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.3] 2024-03-29T10:39:07.436 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA4mgZmM5iZCxAAVOV0AClvBPK9qFZ+LSf3Kg== 2024-03-29T10:39:07.436 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.x] 2024-03-29T10:39:07.436 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQArmgZmJ839NRAAJA061nQ2v6/w7V9gV7sUag== 2024-03-29T10:39:07.436 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.4] 2024-03-29T10:39:07.436 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA+mgZmL/xjGBAAYD/zSdP2ETLpONUpZcA1Kw== 2024-03-29T10:39:07.437 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.5] 2024-03-29T10:39:07.437 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQBBmgZmUAndFhAA5aKa0/PiCMr1+HmPsWuZ9g== 2024-03-29T10:39:07.437 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.6] 2024-03-29T10:39:07.437 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQBEmgZmxaFLMxAADDwh2MpfL9GK5I4lJ8eLwg== 2024-03-29T10:39:07.437 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.7] 2024-03-29T10:39:07.437 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQBHmgZmXtQ3MhAAv+yny/dPVag0m0xQINPRRg== 2024-03-29T10:39:07.437 INFO:teuthology.orchestra.run.smithi176.stdout:[client.0] 2024-03-29T10:39:07.437 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQArmgZmUxGoOBAAs4QZnHcA7lkCfYmoNQiQvQ== 2024-03-29T10:39:07.437 INFO:teuthology.orchestra.run.smithi176.stdout:[client.1] 2024-03-29T10:39:07.437 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAsmgZmWbp5ABAA4xvyNc4usdjn7hAm5SPQeQ== 2024-03-29T10:39:07.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:39:07.458 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:39:07.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.533 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.588 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.642 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:39:07.749 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:39:07.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.857 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.910 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:07.965 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:08.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:39:08.018 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:39:08.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:39:08.075 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:39:08.129 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:39:08.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:39:08.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:39:08.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:39:08.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:39:08.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:39:08.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:39:08.470 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:39:08.499 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:39:08.596 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:39:08.623 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:08.626 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:08.677 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:39:08.678 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:39:08.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:39:08.680 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:39:08.680 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:39:08.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:39:08.682 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:39:08.682 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:39:08.682 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:39:08.720 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:39:08.720 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:39:08.721 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:39:08.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:39:08.723 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:39:08.723 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:39:08.723 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:39:08.725 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:39:08.726 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:39:08.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:39:08.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.842+0000 7fdc898b3640 10 monclient: get_monmap_and_config 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.842+0000 7fdc898b3640 10 monclient: build_initial_monmap 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.842+0000 7fdc898b3640 10 monclient: monmap: 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:39:08.843845+0000 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:39:08.843845+0000 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:08.827 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:08.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 1 Processor -- start 2024-03-29T10:39:08.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 1 -- start start 2024-03-29T10:39:08.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 10 monclient: init 2024-03-29T10:39:08.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:08.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:08.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 10 monclient(hunting): picked mon.noname-b con 0x7fdc8409ac40 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:08.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 10 monclient(hunting): picked mon.noname-c con 0x7fdc8409c870 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:08.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 10 monclient(hunting): picked mon.noname-a con 0x7fdc840b4360 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:08.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8409dec0 con 0x7fdc840b4360 2024-03-29T10:39:08.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8409da30 con 0x7fdc8409c870 2024-03-29T10:39:08.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc840b39c0 con 0x7fdc8409ac40 2024-03-29T10:39:08.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:08.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.843+0000 7fdc898b3640 10 monclient(hunting): authenticate will time out at 814.084990s 2024-03-29T10:39:08.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.844+0000 7fdc82ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fdc8409ac40 0x7fdc840b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42196/0 (socket says 172.21.15.114:42196) 2024-03-29T10:39:08.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.844+0000 7fdc82ffd640 1 -- 172.21.15.114:0/561762745 learned_addr learned my addr 172.21.15.114:0/561762745 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:08.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:08.844+0000 7fdc837fe640 1 --1- 172.21.15.114:0/561762745 >> v1:172.21.15.114:6789/0 conn(0x7fdc840b4360 0x7fdc8414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37884/0 (socket says 172.21.15.114:37884) 2024-03-29T10:39:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 10 monclient(hunting): tick 2024-03-29T10:39:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:39:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 1 monclient(hunting): continuing hunt 2024-03-29T10:39:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:39:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 1 -- 172.21.15.114:0/561762745 >> v1:172.21.15.114:6790/0 conn(0x7fdc8409ac40 legacy=0x7fdc840b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 1 -- 172.21.15.114:0/561762745 >> v1:172.21.15.176:6789/0 conn(0x7fdc8409c870 legacy=0x7fdc8413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 1 -- 172.21.15.114:0/561762745 >> v1:172.21.15.114:6789/0 conn(0x7fdc840b4360 legacy=0x7fdc8414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 10 monclient(hunting): picked mon.noname-a con 0x7fdc640036e0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 10 monclient(hunting): picked mon.noname-b con 0x7fdc8409ac40 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 10 monclient(hunting): picked mon.noname-c con 0x7fdc6400ab30 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 1 -- 172.21.15.114:0/561762745 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc840b5160 con 0x7fdc640036e0 2024-03-29T10:39:11.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 1 -- 172.21.15.114:0/561762745 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6400f820 con 0x7fdc6400ab30 2024-03-29T10:39:11.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 1 -- 172.21.15.114:0/561762745 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc64010a80 con 0x7fdc8409ac40 2024-03-29T10:39:11.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:11.844+0000 7fdc817fa640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:13.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.011+0000 7fdc827fc640 1 -- 172.21.15.114:0/561762745 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257612148 0 0) 0x7fdc840b5160 con 0x7fdc640036e0 2024-03-29T10:39:13.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.011+0000 7fdc827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:13.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.011+0000 7fdc827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:13.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.011+0000 7fdc827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:13.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.011+0000 7fdc827fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:39:13.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.011+0000 7fdc827fc640 1 -- 172.21.15.114:0/561762745 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc70003520 con 0x7fdc640036e0 2024-03-29T10:39:13.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 1 -- 172.21.15.114:0/561762745 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153020105 0 0) 0x7fdc6400f820 con 0x7fdc6400ab30 2024-03-29T10:39:13.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:13.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:13.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:13.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:39:13.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 1 -- 172.21.15.114:0/561762745 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc840b5160 con 0x7fdc6400ab30 2024-03-29T10:39:13.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 1 -- 172.21.15.114:0/561762745 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3722893712 0 0) 0x7fdc70003520 con 0x7fdc640036e0 2024-03-29T10:39:13.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 1 -- 172.21.15.114:0/561762745 >> v1:172.21.15.114:6790/0 conn(0x7fdc8409ac40 legacy=0x7fdc64009350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:13.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 1 -- 172.21.15.114:0/561762745 >> v1:172.21.15.176:6789/0 conn(0x7fdc6400ab30 legacy=0x7fdc6400cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 1 monclient: found mon.noname-a 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 1 -- 172.21.15.114:0/561762745 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc64011ce0 con 0x7fdc640036e0 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 1 -- 172.21.15.114:0/561762745 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdc74001650 con 0x7fdc640036e0 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient: dump: 2024-03-29T10:39:14.000 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc898b3640 5 monclient: authenticate success, global_id 4101 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc898b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 1 -- 172.21.15.114:0/561762745 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc74001820 con 0x7fdc640036e0 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:14.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 1 -- 172.21.15.114:0/561762745 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdc74001bf0 con 0x7fdc640036e0 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc827fc640 10 monclient: dump: 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.002 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc898b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.013+0000 7fdc898b3640 4 set_mon_vals no callback set 2024-03-29T10:39:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.014+0000 7fdc898b3640 10 monclient: shutdown 2024-03-29T10:39:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.014+0000 7fdc898b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.014+0000 7fdc898b3640 1 -- 172.21.15.114:0/561762745 >> v1:172.21.15.114:6789/0 conn(0x7fdc640036e0 legacy=0x7fdc64005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.014+0000 7fdc898b3640 1 -- 172.21.15.114:0/561762745 shutdown_connections 2024-03-29T10:39:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.014+0000 7fdc898b3640 1 -- 172.21.15.114:0/561762745 >> 172.21.15.114:0/561762745 conn(0x7fdc84066b00 msgr2=0x7fdc84066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.014+0000 7fdc898b3640 1 -- 172.21.15.114:0/561762745 shutdown_connections 2024-03-29T10:39:14.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.014+0000 7fdc898b3640 1 -- 172.21.15.114:0/561762745 wait complete. 2024-03-29T10:39:14.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.015+0000 7fdc898b3640 10 monclient: build_initial_monmap 2024-03-29T10:39:14.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.015+0000 7fdc898b3640 10 monclient: monmap: 2024-03-29T10:39:14.008 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-03-29T10:39:14.008 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:14.008 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.015+0000 7fdc898b3640 1 librados: starting msgr at 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.015+0000 7fdc898b3640 1 librados: starting objecter 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.015+0000 7fdc898b3640 1 Processor -- start 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 1 -- start start 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 1 librados: setting wanted keys 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 1 librados: calling monclient init 2024-03-29T10:39:14.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 10 monclient: init 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 10 monclient(hunting): picked mon.noname-c con 0x7fdc640036e0 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 10 monclient(hunting): picked mon.noname-b con 0x7fdc6400ab30 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 10 monclient(hunting): picked mon.noname-a con 0x7fdc8409ac40 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc840ad6f0 con 0x7fdc8409ac40 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc841d75c0 con 0x7fdc6400ab30 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc840ad920 con 0x7fdc640036e0 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.016+0000 7fdc898b3640 10 monclient(hunting): authenticate will time out at 819.257996s 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc82ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fdc640036e0 0x7fdc841d6c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42222/0 (socket says 172.21.15.114:42222) 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc837fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fdc8409ac40 0x7fdc8413ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37924/0 (socket says 172.21.15.114:37924) 2024-03-29T10:39:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc82ffd640 1 -- 172.21.15.114:0/3529720178 learned_addr learned my addr 172.21.15.114:0/3529720178 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc82ffd640 1 -- 172.21.15.114:0/3529720178 >> v1:172.21.15.114:6790/0 conn(0x7fdc640036e0 legacy=0x7fdc841d6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc82ffd640 1 -- 172.21.15.114:0/3529720178 >> v1:172.21.15.114:6790/0 conn(0x7fdc640036e0 legacy=0x7fdc841d6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc82ffd640 1 --1- 172.21.15.114:0/3529720178 >> v1:172.21.15.114:6790/0 conn(0x7fdc640036e0 0x7fdc841d6c60 :-1 s=OPENED pgs=16 cs=1 l=1).handle_message read tag failed 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 10 client.?.objecter ms_handle_connect 0x7fdc6400ab30 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc82ffd640 1 --1- 172.21.15.114:0/3529720178 >> v1:172.21.15.114:6790/0 conn(0x7fdc640036e0 0x7fdc841d6c60 :-1 s=OPENED pgs=16 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 10 client.?.objecter ms_handle_connect 0x7fdc640036e0 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.114:6790/0 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 10 client.?.objecter ms_handle_connect 0x7fdc8409ac40 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41844634 0 0) 0x7fdc841d75c0 con 0x7fdc6400ab30 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.017+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc64006a20 con 0x7fdc6400ab30 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3895401185 0 0) 0x7fdc64006a20 con 0x7fdc6400ab30 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc841d75c0 con 0x7fdc6400ab30 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdc6c0017f0 con 0x7fdc6400ab30 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915729630 0 0) 0x7fdc840ad6f0 con 0x7fdc8409ac40 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient(hunting): my global_id is 4110 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc64006a20 con 0x7fdc8409ac40 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4048705661 0 0) 0x7fdc64006a20 con 0x7fdc8409ac40 2024-03-29T10:39:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc840ad6f0 con 0x7fdc8409ac40 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdc7c001ad0 con 0x7fdc8409ac40 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.014 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3805748735 0 0) 0x7fdc840ad6f0 con 0x7fdc8409ac40 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 >> v1:172.21.15.114:6790/0 conn(0x7fdc640036e0 legacy=0x7fdc841d6c60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 >> v1:172.21.15.176:6789/0 conn(0x7fdc6400ab30 legacy=0x7fdc8413aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 monclient: found mon.a 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc841d7a50 con 0x7fdc8409ac40 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient: _finish_auth 0 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc898b3640 5 monclient: authenticate success, global_id 4110 2024-03-29T10:39:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc898b3640 10 monclient: _renew_subs 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc898b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.018+0000 7fdc898b3640 1 -- 172.21.15.114:0/3529720178 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8413b5b0 con 0x7fdc8409ac40 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc898b3640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc898b3640 10 monclient: _renew_subs 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc898b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc898b3640 1 -- 172.21.15.114:0/3529720178 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc84140890 con 0x7fdc8409ac40 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc898b3640 1 librados: init done 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc7c001f80 con 0x7fdc8409ac40 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdc7c004d00 con 0x7fdc8409ac40 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 10 monclient: dump: 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T10:39:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc81ffb640 4 set_mon_vals no callback set 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdc7c0051f0 con 0x7fdc8409ac40 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1560421715 0 0) 0x7fdc7c006680 con 0x7fdc8409ac40 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 10 client.4110.objecter ms_dispatch 0x7fdc84000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:39:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.019+0000 7fdc80ff9640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T10:39:14.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.020+0000 7fdc898b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:14.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.020+0000 7fdc898b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:14.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.020+0000 7fdc898b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:14.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.020+0000 7fdc898b3640 1 -- 172.21.15.114:0/3529720178 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc68005080 con 0x7fdc8409ac40 2024-03-29T10:39:14.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.023+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdc7c006890 con 0x7fdc8409ac40 2024-03-29T10:39:14.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.023+0000 7fdc80ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:14.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.023+0000 7fdc80ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:14.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.094+0000 7fdc898b3640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:39:14.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.094+0000 7fdc898b3640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:39:14.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.094+0000 7fdc898b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:14.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.094+0000 7fdc898b3640 1 -- 172.21.15.114:0/3529720178 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fdc68005330 con 0x7fdc8409ac40 2024-03-29T10:39:14.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.096+0000 7fdc80ff9640 1 -- 172.21.15.114:0/3529720178 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fdc7c005670 con 0x7fdc8409ac40 2024-03-29T10:39:14.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.096+0000 7fdc80ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:39:14.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.096+0000 7fdc80ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:39:14.080 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-03-29T10:39:14.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 10 librados: watch_flush enter 2024-03-29T10:39:14.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 10 librados: watch_flush exit 2024-03-29T10:39:14.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T10:39:14.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 10 client.4110.objecter successfully canceled tick 2024-03-29T10:39:14.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 10 monclient: shutdown 2024-03-29T10:39:14.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:14.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 1 -- 172.21.15.114:0/3529720178 >> v1:172.21.15.114:6789/0 conn(0x7fdc8409ac40 legacy=0x7fdc8413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 1 -- 172.21.15.114:0/3529720178 shutdown_connections 2024-03-29T10:39:14.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 1 -- 172.21.15.114:0/3529720178 >> 172.21.15.114:0/3529720178 conn(0x7fdc84066b00 msgr2=0x7fdc8413bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:14.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 1 -- 172.21.15.114:0/3529720178 shutdown_connections 2024-03-29T10:39:14.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 1 -- 172.21.15.114:0/3529720178 wait complete. 2024-03-29T10:39:14.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:14.097+0000 7fdc898b3640 1 librados: shutdown 2024-03-29T10:39:14.092 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:39:14.093 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:39:14.093 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:39:14.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:39:14.121 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:39:14.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:39:14.188 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:39:14.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:39:14.259 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T10:39:14.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:39:14.326 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:39:14.326 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:39:14.353 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:39:14.353 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:39:14.420 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:39:14.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:39:14.487 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T10:39:14.487 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:39:14.490 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:14.506+0000 7fbb4763e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:39:14.496 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:14.513+0000 7fdf5662b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:39:14.522 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new b22e9cf8-6612-4468-b6b2-a6c1574f2921 0 2024-03-29T10:39:14.544 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:14.560+0000 7fbb4763e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:39:14.553 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:14.569+0000 7fdf5662b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:39:14.652 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.668+0000 7f25acdc8640 10 monclient: get_monmap_and_config 2024-03-29T10:39:14.652 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.668+0000 7f25acdc8640 10 monclient: build_initial_monmap 2024-03-29T10:39:14.652 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.668+0000 7f25acdc8640 10 monclient: monmap: 2024-03-29T10:39:14.652 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:14.652 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.652 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:39:14.669282+0000 2024-03-29T10:39:14.652 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:39:14.669282+0000 2024-03-29T10:39:14.652 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:14.653 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.653 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:14.653 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:14.653 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:14.653 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.668+0000 7f25acdc8640 1 Processor -- start 2024-03-29T10:39:14.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.668+0000 7f25acdc8640 1 -- start start 2024-03-29T10:39:14.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.668+0000 7f25acdc8640 10 monclient: init 2024-03-29T10:39:14.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25acdc8640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:14.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25acdc8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:14.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25acdc8640 10 monclient(hunting): picked mon.noname-c con 0x7f25a80a9cc0 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:14.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25acdc8640 10 monclient(hunting): picked mon.noname-b con 0x7f25a80aae80 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:14.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25acdc8640 10 monclient(hunting): picked mon.noname-a con 0x7f25a80aa0a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:14.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25acdc8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80abc30 con 0x7f25a80aa0a0 2024-03-29T10:39:14.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25acdc8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80abf10 con 0x7f25a80a9cc0 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25acdc8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80aa960 con 0x7f25a80aae80 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25acdc8640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25acdc8640 10 monclient(hunting): authenticate will time out at 822.953699s 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25a659c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f25a80a9cc0 0x7f25a80ac610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59222/0 (socket says 172.21.15.176:59222) 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.669+0000 7f25a659c640 1 -- 172.21.15.176:0/3151293961 learned_addr learned my addr 172.21.15.176:0/3151293961 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429547957 0 0) 0x7f25a80abf10 con 0x7f25a80a9cc0 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590004630 con 0x7f25a80a9cc0 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131347119 0 0) 0x7f25a80aa960 con 0x7f25a80aae80 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.663 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): my global_id is 4109 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a80abf10 con 0x7f25a80aae80 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519114255 0 0) 0x7f25a80abc30 con 0x7f25a80aa0a0 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a80aa960 con 0x7f25a80aa0a0 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1430486636 0 0) 0x7f2590004630 con 0x7f25a80a9cc0 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 >> v1:172.21.15.114:6790/0 conn(0x7f25a80aae80 legacy=0x7f25a813e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 >> v1:172.21.15.114:6789/0 conn(0x7f25a80aa0a0 legacy=0x7f25a8149db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient: _finish_hunting 0 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 monclient: found mon.noname-c 2024-03-29T10:39:14.664 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a80ada90 con 0x7f25a80a9cc0 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient: _finish_auth 0 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient: _check_auth_tickets 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2598001690 con 0x7f25a80a9cc0 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25acdc8640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25acdc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient: dump: 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.665 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25acdc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2598001870 con 0x7f25a80a9cc0 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25acdc8640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:14.666 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 1 -- 172.21.15.176:0/3151293961 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2598001c80 con 0x7f25a80a9cc0 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25acdc8640 4 set_mon_vals no callback set 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25a559a640 10 monclient: dump: 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.667 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25acdc8640 10 monclient: shutdown 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25acdc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.670+0000 7f25acdc8640 1 -- 172.21.15.176:0/3151293961 >> v1:172.21.15.176:6789/0 conn(0x7f25a80a9cc0 legacy=0x7f25a80ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.671+0000 7f25acdc8640 1 -- 172.21.15.176:0/3151293961 shutdown_connections 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.671+0000 7f25acdc8640 1 -- 172.21.15.176:0/3151293961 >> 172.21.15.176:0/3151293961 conn(0x7f25a80b3750 msgr2=0x7f25a80b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.671+0000 7f25acdc8640 1 -- 172.21.15.176:0/3151293961 shutdown_connections 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.671+0000 7f25acdc8640 1 -- 172.21.15.176:0/3151293961 wait complete. 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.671+0000 7f25acdc8640 10 monclient: build_initial_monmap 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.671+0000 7f25acdc8640 10 monclient: monmap: 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T10:39:14.668 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 1 librados: starting msgr at 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 1 librados: starting objecter 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 1 Processor -- start 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 1 -- start start 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 1 librados: setting wanted keys 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 1 librados: calling monclient init 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 10 monclient: init 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:14.669 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 10 monclient(hunting): picked mon.noname-b con 0x7f25a80a9cc0 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 10 monclient(hunting): picked mon.noname-c con 0x7f25a80aa0a0 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 10 monclient(hunting): picked mon.noname-a con 0x7f25a80aae80 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a81dac70 con 0x7f25a80aae80 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a813b7a0 con 0x7f25a80a9cc0 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a81d7720 con 0x7f25a80aa0a0 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25acdc8640 10 monclient(hunting): authenticate will time out at 822.956930s 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25a659c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f25a80a9cc0 0x7f25a81d7010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59232/0 (socket says 172.21.15.176:59232) 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.672+0000 7f25a659c640 1 -- 172.21.15.176:0/855960453 learned_addr learned my addr 172.21.15.176:0/855960453 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 client.?.objecter ms_handle_connect 0x7f25a80a9cc0 2024-03-29T10:39:14.670 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 client.?.objecter ms_handle_connect 0x7f25a80aae80 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226612517 0 0) 0x7f25a813b7a0 con 0x7f25a80a9cc0 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2588004eb0 con 0x7f25a80a9cc0 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 client.?.objecter ms_handle_connect 0x7f25a80aa0a0 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316340681 0 0) 0x7f25a81dac70 con 0x7f25a80aae80 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.671 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a813b7a0 con 0x7f25a80aae80 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611739090 0 0) 0x7f25a81d7720 con 0x7f25a80aa0a0 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a81dac70 con 0x7f25a80aa0a0 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3692640918 0 0) 0x7f2588004eb0 con 0x7f25a80a9cc0 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25a81d7720 con 0x7f25a80a9cc0 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2598001bc0 con 0x7f25a80a9cc0 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:14.672 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3725340025 0 0) 0x7f25a813b7a0 con 0x7f25a80aae80 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2588004eb0 con 0x7f25a80aae80 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f25a0001bf0 con 0x7f25a80aae80 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.673 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4145541720 0 0) 0x7f25a81dac70 con 0x7f25a80aa0a0 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25a813b7a0 con 0x7f25a80aa0a0 2024-03-29T10:39:14.674 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f259c001770 con 0x7f25a80aa0a0 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3138008608 0 0) 0x7f25a81d7720 con 0x7f25a80a9cc0 2024-03-29T10:39:14.675 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 >> v1:172.21.15.114:6790/0 conn(0x7f25a80aa0a0 legacy=0x7f25a813a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 >> v1:172.21.15.114:6789/0 conn(0x7f25a80aae80 legacy=0x7f25a813b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 monclient: found mon.b 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a813b9e0 con 0x7f25a80a9cc0 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f25acdc8640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f25acdc8640 10 monclient: _renew_subs 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f25acdc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f25acdc8640 1 -- 172.21.15.176:0/855960453 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a813bc90 con 0x7f25a80a9cc0 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f25acdc8640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f25acdc8640 10 monclient: _renew_subs 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f25acdc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f25acdc8640 1 -- 172.21.15.176:0/855960453 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a81db2e0 con 0x7f25a80a9cc0 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f25acdc8640 1 librados: init done 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2598001170 con 0x7f25a80a9cc0 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2598004d90 con 0x7f25a80a9cc0 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 monclient: dump: 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.677 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f25980052c0 con 0x7f25a80a9cc0 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1017537221 0 0) 0x7f2598006830 con 0x7f25a80a9cc0 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f25a4d99640 4 set_mon_vals no callback set 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 10 client.4117.objecter ms_dispatch 0x7f25a8000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:39:14.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 3 client.4117.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:39:14.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.673+0000 7f2596ffd640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T10:39:14.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.674+0000 7f25acdc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:14.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.674+0000 7f25acdc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:14.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.674+0000 7f25acdc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.674+0000 7f25acdc8640 1 -- 172.21.15.176:0/855960453 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258c005080 con 0x7f25a80a9cc0 2024-03-29T10:39:14.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.676+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2598006b10 con 0x7f25a80a9cc0 2024-03-29T10:39:14.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.676+0000 7f2596ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:14.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.676+0000 7f2596ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:14.732 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.748+0000 7f25acdc8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b22e9cf8-6612-4468-b6b2-a6c1574f2921", "id": 0}] 2024-03-29T10:39:14.732 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.748+0000 7f25acdc8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b22e9cf8-6612-4468-b6b2-a6c1574f2921", "id": 0}] 2024-03-29T10:39:14.732 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.748+0000 7f25acdc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.732 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.748+0000 7f25acdc8640 1 -- 172.21.15.176:0/855960453 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b22e9cf8-6612-4468-b6b2-a6c1574f2921", "id": 0} v 0) v1 -- 0x7f258c0058b0 con 0x7f25a80a9cc0 2024-03-29T10:39:14.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.753+0000 7f2596ffd640 1 -- 172.21.15.176:0/855960453 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b22e9cf8-6612-4468-b6b2-a6c1574f2921", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3302179874 0 732707515) 0x7f2598005920 con 0x7f25a80a9cc0 2024-03-29T10:39:14.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.753+0000 7f2596ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b22e9cf8-6612-4468-b6b2-a6c1574f2921", "id": 0}] 2024-03-29T10:39:14.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.753+0000 7f2596ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:14.737 INFO:teuthology.orchestra.run.smithi176.stdout:0 2024-03-29T10:39:14.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 10 librados: watch_flush enter 2024-03-29T10:39:14.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 10 librados: watch_flush exit 2024-03-29T10:39:14.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T10:39:14.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 10 client.4117.objecter successfully canceled tick 2024-03-29T10:39:14.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 10 monclient: shutdown 2024-03-29T10:39:14.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:14.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 1 -- 172.21.15.176:0/855960453 >> v1:172.21.15.176:6789/0 conn(0x7f25a80a9cc0 legacy=0x7f25a81d7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 1 -- 172.21.15.176:0/855960453 shutdown_connections 2024-03-29T10:39:14.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 1 -- 172.21.15.176:0/855960453 >> 172.21.15.176:0/855960453 conn(0x7f25a80b3750 msgr2=0x7f25a80b2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:14.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 1 -- 172.21.15.176:0/855960453 shutdown_connections 2024-03-29T10:39:14.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 1 -- 172.21.15.176:0/855960453 wait complete. 2024-03-29T10:39:14.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.754+0000 7f25acdc8640 1 librados: shutdown 2024-03-29T10:39:14.748 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new d8ca84b0-980e-461e-962f-0bcb7e2cf005 1 2024-03-29T10:39:14.848 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.863+0000 7f9b8355d640 10 monclient: get_monmap_and_config 2024-03-29T10:39:14.848 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.863+0000 7f9b8355d640 10 monclient: build_initial_monmap 2024-03-29T10:39:14.848 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.863+0000 7f9b8355d640 10 monclient: monmap: 2024-03-29T10:39:14.848 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:14.848 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.848 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:39:14.864804+0000 2024-03-29T10:39:14.848 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:39:14.864804+0000 2024-03-29T10:39:14.848 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:14.848 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.849 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:14.849 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:14.849 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:14.849 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.849 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.864+0000 7f9b8355d640 1 Processor -- start 2024-03-29T10:39:14.858 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.864+0000 7f9b8355d640 1 -- start start 2024-03-29T10:39:14.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.864+0000 7f9b8355d640 10 monclient: init 2024-03-29T10:39:14.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:14.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:14.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b7c09ac40 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:14.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7c09c870 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:14.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7c0b4360 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:14.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7c09dec0 con 0x7f9b7c09ac40 2024-03-29T10:39:14.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7c09da30 con 0x7f9b7c0b4360 2024-03-29T10:39:14.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7c0b39c0 con 0x7f9b7c09c870 2024-03-29T10:39:14.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:14.859 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 10 monclient(hunting): authenticate will time out at 823.149225s 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b81afa640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9b7c0b4360 0x7f9b7c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59234/0 (socket says 172.21.15.176:59234) 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b81afa640 1 -- 172.21.15.176:0/368429016 learned_addr learned my addr 172.21.15.176:0/368429016 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407818043 0 0) 0x7f9b7c09da30 con 0x7f9b7c0b4360 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b68003520 con 0x7f9b7c0b4360 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848828567 0 0) 0x7f9b6c001110 con 0x7f9b7c09ac40 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.860 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b7c09da30 con 0x7f9b7c09ac40 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107704665 0 0) 0x7f9b7c0b39c0 con 0x7f9b7c09c870 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient(hunting): my global_id is 4115 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b6c001110 con 0x7f9b7c09c870 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4192917175 0 0) 0x7f9b68003520 con 0x7f9b7c0b4360 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 >> v1:172.21.15.114:6790/0 conn(0x7f9b7c09c870 legacy=0x7f9b7c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 >> v1:172.21.15.114:6789/0 conn(0x7f9b7c09ac40 legacy=0x7f9b7c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 monclient: found mon.noname-c 2024-03-29T10:39:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7c0b5160 con 0x7f9b7c0b4360 2024-03-29T10:39:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9b74001990 con 0x7f9b7c0b4360 2024-03-29T10:39:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:14.862 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient: dump: 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:14.863 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b74001af0 con 0x7f9b7c0b4360 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 1 -- 172.21.15.176:0/368429016 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9b74001f40 con 0x7f9b7c0b4360 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b8355d640 4 set_mon_vals no callback set 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.865+0000 7f9b73fff640 10 monclient: dump: 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.864 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.866+0000 7f9b8355d640 10 monclient: shutdown 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.866+0000 7f9b8355d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.866+0000 7f9b8355d640 1 -- 172.21.15.176:0/368429016 >> v1:172.21.15.176:6789/0 conn(0x7f9b7c0b4360 legacy=0x7f9b7c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.866+0000 7f9b8355d640 1 -- 172.21.15.176:0/368429016 shutdown_connections 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.866+0000 7f9b8355d640 1 -- 172.21.15.176:0/368429016 >> 172.21.15.176:0/368429016 conn(0x7f9b7c066b00 msgr2=0x7f9b7c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.866+0000 7f9b8355d640 1 -- 172.21.15.176:0/368429016 shutdown_connections 2024-03-29T10:39:14.865 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.866+0000 7f9b8355d640 1 -- 172.21.15.176:0/368429016 wait complete. 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.867+0000 7f9b8355d640 10 monclient: build_initial_monmap 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.867+0000 7f9b8355d640 10 monclient: monmap: 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:14.866 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.867+0000 7f9b8355d640 1 librados: starting msgr at 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.867+0000 7f9b8355d640 1 librados: starting objecter 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.867+0000 7f9b8355d640 1 Processor -- start 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.867+0000 7f9b8355d640 1 -- start start 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.867+0000 7f9b8355d640 1 librados: setting wanted keys 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.867+0000 7f9b8355d640 1 librados: calling monclient init 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.867+0000 7f9b8355d640 10 monclient: init 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.867+0000 7f9b8355d640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7c09ac40 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b7c141820 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7c0b06e0 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:14.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7c143170 con 0x7f9b7c141820 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7c1433e0 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7c141dd0 con 0x7f9b7c09ac40 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 10 monclient(hunting): authenticate will time out at 823.152120s 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b81afa640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9b7c0b06e0 0x7f9b7c138ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59238/0 (socket says 172.21.15.176:59238) 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b81afa640 1 -- 172.21.15.176:0/2948425709 learned_addr learned my addr 172.21.15.176:0/2948425709 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 client.?.objecter ms_handle_connect 0x7f9b7c0b06e0 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 client.?.objecter ms_handle_connect 0x7f9b7c09ac40 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 client.?.objecter ms_handle_connect 0x7f9b7c141820 2024-03-29T10:39:14.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987558332 0 0) 0x7f9b7c1433e0 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b60004eb0 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882621848 0 0) 0x7f9b7c141dd0 con 0x7f9b7c09ac40 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:39:14.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b7c1433e0 con 0x7f9b7c09ac40 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527311150 0 0) 0x7f9b7c143170 con 0x7f9b7c141820 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b7c141dd0 con 0x7f9b7c141820 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2051212697 0 0) 0x7f9b60004eb0 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b7c143170 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9b74001e40 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:14.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 644167760 0 0) 0x7f9b7c1433e0 con 0x7f9b7c09ac40 2024-03-29T10:39:14.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b60004eb0 con 0x7f9b7c09ac40 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9b6c001920 con 0x7f9b7c09ac40 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.872 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1070244660 0 0) 0x7f9b7c141dd0 con 0x7f9b7c141820 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b7c1433e0 con 0x7f9b7c141820 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9b78001870 con 0x7f9b7c141820 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.873 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.874 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.874 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.874 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.874 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.874 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.874 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 587464738 0 0) 0x7f9b7c143170 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 >> v1:172.21.15.114:6790/0 conn(0x7f9b7c09ac40 legacy=0x7f9b7c140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 >> v1:172.21.15.114:6789/0 conn(0x7f9b7c141820 legacy=0x7f9b7c0affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:14.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 monclient: found mon.b 2024-03-29T10:39:14.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:14.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7c1422f0 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:14.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:14.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:14.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:39:14.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 10 monclient: _renew_subs 2024-03-29T10:39:14.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b8355d640 1 -- 172.21.15.176:0/2948425709 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b7c1395b0 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b74002080 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9b74004fb0 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b71ffb640 10 monclient: dump: 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:14.876 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.868+0000 7f9b737fe640 4 set_mon_vals no callback set 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b8355d640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b8355d640 10 monclient: _renew_subs 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9b74005520 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b8355d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b8355d640 1 -- 172.21.15.176:0/2948425709 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7c1e3cd0 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b8355d640 1 librados: init done 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2991420250 0 0) 0x7f9b74005b80 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b71ffb640 10 client.4123.objecter ms_dispatch 0x7f9b7c000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b71ffb640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b71ffb640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b71ffb640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b8355d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b8355d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b8355d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.869+0000 7f9b8355d640 1 -- 172.21.15.176:0/2948425709 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7c141f60 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.871+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9b74005d40 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.871+0000 7f9b71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.871+0000 7f9b71ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:14.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.944+0000 7f9b8355d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d8ca84b0-980e-461e-962f-0bcb7e2cf005", "id": 1}] 2024-03-29T10:39:14.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.944+0000 7f9b8355d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d8ca84b0-980e-461e-962f-0bcb7e2cf005", "id": 1}] 2024-03-29T10:39:14.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.944+0000 7f9b8355d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:14.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.944+0000 7f9b8355d640 1 -- 172.21.15.176:0/2948425709 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d8ca84b0-980e-461e-962f-0bcb7e2cf005", "id": 1} v 0) v1 -- 0x7f9b7c09b020 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.933 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.949+0000 7f9b71ffb640 1 -- 172.21.15.176:0/2948425709 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d8ca84b0-980e-461e-962f-0bcb7e2cf005", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 4153720305 0 940483276) 0x7f9b74006110 con 0x7f9b7c0b06e0 2024-03-29T10:39:14.933 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.949+0000 7f9b71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d8ca84b0-980e-461e-962f-0bcb7e2cf005", "id": 1}] 2024-03-29T10:39:14.933 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.949+0000 7f9b71ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:14.934 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-03-29T10:39:14.935 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.950+0000 7f9b8355d640 10 librados: watch_flush enter 2024-03-29T10:39:14.935 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.950+0000 7f9b8355d640 10 librados: watch_flush exit 2024-03-29T10:39:14.935 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.950+0000 7f9b8355d640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T10:39:14.935 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.950+0000 7f9b8355d640 10 client.4123.objecter successfully canceled tick 2024-03-29T10:39:14.935 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.950+0000 7f9b8355d640 10 monclient: shutdown 2024-03-29T10:39:14.935 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.950+0000 7f9b8355d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:14.935 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.950+0000 7f9b8355d640 1 -- 172.21.15.176:0/2948425709 >> v1:172.21.15.176:6789/0 conn(0x7f9b7c0b06e0 legacy=0x7f9b7c138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:14.935 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.950+0000 7f9b8355d640 1 -- 172.21.15.176:0/2948425709 shutdown_connections 2024-03-29T10:39:14.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.950+0000 7f9b8355d640 1 -- 172.21.15.176:0/2948425709 >> 172.21.15.176:0/2948425709 conn(0x7f9b7c066b00 msgr2=0x7f9b7c0b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:14.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.950+0000 7f9b8355d640 1 -- 172.21.15.176:0/2948425709 shutdown_connections 2024-03-29T10:39:14.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.950+0000 7f9b8355d640 1 -- 172.21.15.176:0/2948425709 wait complete. 2024-03-29T10:39:14.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:14.951+0000 7f9b8355d640 1 librados: shutdown 2024-03-29T10:39:14.945 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new d83b07bd-dc3c-4266-b151-9bd1e1bf252d 2 2024-03-29T10:39:15.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.063+0000 7ff705886640 10 monclient: get_monmap_and_config 2024-03-29T10:39:15.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.063+0000 7ff705886640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.063+0000 7ff705886640 10 monclient: monmap: 2024-03-29T10:39:15.047 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:15.048 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.048 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:39:15.064474+0000 2024-03-29T10:39:15.048 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:39:15.064474+0000 2024-03-29T10:39:15.048 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.048 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.048 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:15.048 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:15.048 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:15.048 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 1 Processor -- start 2024-03-29T10:39:15.056 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 1 -- start start 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 10 monclient: init 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 10 monclient(hunting): picked mon.noname-c con 0x7ff7000b8240 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 10 monclient(hunting): picked mon.noname-a con 0x7ff7000b1f60 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 10 monclient(hunting): picked mon.noname-b con 0x7ff70009c8c0 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70009bd80 con 0x7ff7000b1f60 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70009b960 con 0x7ff7000b8240 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7000b2ec0 con 0x7ff70009c8c0 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff705886640 10 monclient(hunting): authenticate will time out at 823.348749s 2024-03-29T10:39:15.057 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff6feffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ff7000b8240 0x7ff70013f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59244/0 (socket says 172.21.15.176:59244) 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.064+0000 7ff6feffd640 1 -- 172.21.15.176:0/4124754369 learned_addr learned my addr 172.21.15.176:0/4124754369 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434695636 0 0) 0x7ff70009b960 con 0x7ff7000b8240 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e8003520 con 0x7ff7000b8240 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707615335 0 0) 0x7ff70009bd80 con 0x7ff7000b1f60 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff70009b960 con 0x7ff7000b1f60 2024-03-29T10:39:15.058 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759307554 0 0) 0x7ff7000b2ec0 con 0x7ff70009c8c0 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff70009bd80 con 0x7ff70009c8c0 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 822917589 0 0) 0x7ff6e8003520 con 0x7ff7000b8240 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 >> v1:172.21.15.114:6790/0 conn(0x7ff70009c8c0 legacy=0x7ff70014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 >> v1:172.21.15.114:6789/0 conn(0x7ff7000b1f60 legacy=0x7ff70009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 monclient: found mon.noname-c 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70013fb60 con 0x7ff7000b8240 2024-03-29T10:39:15.059 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff6f0001690 con 0x7ff7000b8240 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient: dump: 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.060 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff705886640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff705886640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6f0001870 con 0x7ff7000b8240 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 1 -- 172.21.15.176:0/4124754369 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff6f0001c80 con 0x7ff7000b8240 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff705886640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff705886640 4 set_mon_vals no callback set 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff6fdffb640 10 monclient: dump: 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.061 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff705886640 10 monclient: shutdown 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff705886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff705886640 1 -- 172.21.15.176:0/4124754369 >> v1:172.21.15.176:6789/0 conn(0x7ff7000b8240 legacy=0x7ff70013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff705886640 1 -- 172.21.15.176:0/4124754369 shutdown_connections 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff705886640 1 -- 172.21.15.176:0/4124754369 >> 172.21.15.176:0/4124754369 conn(0x7ff700066b00 msgr2=0x7ff700066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff705886640 1 -- 172.21.15.176:0/4124754369 shutdown_connections 2024-03-29T10:39:15.062 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.065+0000 7ff705886640 1 -- 172.21.15.176:0/4124754369 wait complete. 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.066+0000 7ff705886640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.066+0000 7ff705886640 10 monclient: monmap: 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.066+0000 7ff705886640 1 librados: starting msgr at 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.066+0000 7ff705886640 1 librados: starting objecter 2024-03-29T10:39:15.063 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.066+0000 7ff705886640 1 Processor -- start 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 1 -- start start 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 1 librados: setting wanted keys 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 1 librados: calling monclient init 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 10 monclient: init 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 10 monclient(hunting): picked mon.noname-a con 0x7ff70009c8c0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 10 monclient(hunting): picked mon.noname-c con 0x7ff7000b8240 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 10 monclient(hunting): picked mon.noname-b con 0x7ff700131e10 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff700133f20 con 0x7ff70009c8c0 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff700132de0 con 0x7ff700131e10 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff700134620 con 0x7ff7000b8240 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.064 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff705886640 10 monclient(hunting): authenticate will time out at 823.351628s 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6ff7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ff700131e10 0x7ff7001e1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59258/0 (socket says 172.21.15.176:59258) 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6ff7fe640 1 -- 172.21.15.176:0/2021200206 learned_addr learned my addr 172.21.15.176:0/2021200206 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 client.?.objecter ms_handle_connect 0x7ff700131e10 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 client.?.objecter ms_handle_connect 0x7ff7000b8240 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 client.?.objecter ms_handle_connect 0x7ff70009c8c0 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422755585 0 0) 0x7ff700132de0 con 0x7ff700131e10 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6ec004eb0 con 0x7ff700131e10 2024-03-29T10:39:15.065 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372175423 0 0) 0x7ff700133f20 con 0x7ff70009c8c0 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.067+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff700132de0 con 0x7ff70009c8c0 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560211547 0 0) 0x7ff700134620 con 0x7ff7000b8240 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff700133f20 con 0x7ff7000b8240 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2515558015 0 0) 0x7ff6ec004eb0 con 0x7ff700131e10 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff700134620 con 0x7ff700131e10 2024-03-29T10:39:15.066 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff6f8001d10 con 0x7ff700131e10 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): dump: 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:15.067 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 472806806 0 0) 0x7ff700132de0 con 0x7ff70009c8c0 2024-03-29T10:39:15.068 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6ec004eb0 con 0x7ff70009c8c0 2024-03-29T10:39:15.068 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff6f0001b80 con 0x7ff70009c8c0 2024-03-29T10:39:15.068 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.068 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:15.068 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): dump: 2024-03-29T10:39:15.068 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.068 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.068 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.068 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.068 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.068 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.069 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.069 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.069 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.069 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.069 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1377572223 0 0) 0x7ff700133f20 con 0x7ff7000b8240 2024-03-29T10:39:15.069 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff700132de0 con 0x7ff7000b8240 2024-03-29T10:39:15.069 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff6f4001930 con 0x7ff7000b8240 2024-03-29T10:39:15.069 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.069 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:15.069 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient(hunting): dump: 2024-03-29T10:39:15.069 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 469575169 0 0) 0x7ff700134620 con 0x7ff700131e10 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 >> v1:172.21.15.114:6790/0 conn(0x7ff7000b8240 legacy=0x7ff7001d7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 >> v1:172.21.15.114:6789/0 conn(0x7ff70009c8c0 legacy=0x7ff7001326d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 monclient: found mon.b 2024-03-29T10:39:15.070 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7001d7b80 con 0x7ff700131e10 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff705886640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff705886640 10 monclient: _renew_subs 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff705886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff705886640 1 -- 172.21.15.176:0/2021200206 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7001d7e30 con 0x7ff700131e10 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff705886640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff705886640 10 monclient: _renew_subs 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff705886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff705886640 1 -- 172.21.15.176:0/2021200206 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7001e3440 con 0x7ff700131e10 2024-03-29T10:39:15.071 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff705886640 1 librados: init done 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6f8001ff0 con 0x7ff700131e10 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff6f8003de0 con 0x7ff700131e10 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6fd7fa640 4 set_mon_vals no callback set 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 monclient: dump: 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.072 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff6f8004390 con 0x7ff700131e10 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2725478046 0 0) 0x7ff6f8006a40 con 0x7ff700131e10 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 10 client.4129.objecter ms_dispatch 0x7ff700000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.068+0000 7ff6e7fff640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.069+0000 7ff705886640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.069+0000 7ff705886640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.069+0000 7ff705886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.069+0000 7ff705886640 1 -- 172.21.15.176:0/2021200206 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e0005080 con 0x7ff700131e10 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.071+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff6f80065a0 con 0x7ff700131e10 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.071+0000 7ff6e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.071+0000 7ff6e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:15.105 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:15.121+0000 7fbb4763e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:39:15.108 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:15.125+0000 7fdf5662b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:39:15.146 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.162+0000 7ff705886640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d83b07bd-dc3c-4266-b151-9bd1e1bf252d", "id": 2}] 2024-03-29T10:39:15.146 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.162+0000 7ff705886640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d83b07bd-dc3c-4266-b151-9bd1e1bf252d", "id": 2}] 2024-03-29T10:39:15.146 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.162+0000 7ff705886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.162+0000 7ff705886640 1 -- 172.21.15.176:0/2021200206 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d83b07bd-dc3c-4266-b151-9bd1e1bf252d", "id": 2} v 0) v1 -- 0x7ff6e00058b0 con 0x7ff700131e10 2024-03-29T10:39:15.154 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.170+0000 7ff6e7fff640 1 -- 172.21.15.176:0/2021200206 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d83b07bd-dc3c-4266-b151-9bd1e1bf252d", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 806896875 0 216599125) 0x7ff6f8006880 con 0x7ff700131e10 2024-03-29T10:39:15.154 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.170+0000 7ff6e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d83b07bd-dc3c-4266-b151-9bd1e1bf252d", "id": 2}] 2024-03-29T10:39:15.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.170+0000 7ff6e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:15.155 INFO:teuthology.orchestra.run.smithi176.stdout:2 2024-03-29T10:39:15.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 10 librados: watch_flush enter 2024-03-29T10:39:15.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 10 librados: watch_flush exit 2024-03-29T10:39:15.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T10:39:15.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 10 client.4129.objecter successfully canceled tick 2024-03-29T10:39:15.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 10 monclient: shutdown 2024-03-29T10:39:15.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 1 -- 172.21.15.176:0/2021200206 >> v1:172.21.15.176:6789/0 conn(0x7ff700131e10 legacy=0x7ff7001e1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 1 -- 172.21.15.176:0/2021200206 shutdown_connections 2024-03-29T10:39:15.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 1 -- 172.21.15.176:0/2021200206 >> 172.21.15.176:0/2021200206 conn(0x7ff700066b00 msgr2=0x7ff70009c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 1 -- 172.21.15.176:0/2021200206 shutdown_connections 2024-03-29T10:39:15.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 1 -- 172.21.15.176:0/2021200206 wait complete. 2024-03-29T10:39:15.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.171+0000 7ff705886640 1 librados: shutdown 2024-03-29T10:39:15.159 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:15.175+0000 7fbb4763e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:39:15.166 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d 3 2024-03-29T10:39:15.166 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:15.182+0000 7fdf5662b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:39:15.203 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:15.219+0000 7fbb4763e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:39:15.213 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:15.229+0000 7fdf5662b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:39:15.256 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:15.272+0000 7fbb4763e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:39:15.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.282+0000 7f24d5c74640 10 monclient: get_monmap_and_config 2024-03-29T10:39:15.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.282+0000 7f24d5c74640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.282+0000 7f24d5c74640 10 monclient: monmap: 2024-03-29T10:39:15.266 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:15.267 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.267 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:39:15.283510+0000 2024-03-29T10:39:15.267 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:39:15.283510+0000 2024-03-29T10:39:15.267 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.267 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.267 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:15.267 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:15.267 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:15.267 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 1 Processor -- start 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 1 -- start start 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 10 monclient: init 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 10 monclient(hunting): picked mon.noname-c con 0x7f24d00b1780 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 10 monclient(hunting): picked mon.noname-b con 0x7f24d013d7e0 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 10 monclient(hunting): picked mon.noname-a con 0x7f24d0143be0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d0070b30 con 0x7f24d0143be0 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d0070da0 con 0x7f24d00b1780 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d009adb0 con 0x7f24d013d7e0 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.273 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.283+0000 7f24d5c74640 10 monclient(hunting): authenticate will time out at 823.568002s 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24cf7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f24d00b1780 0x7f24d013d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59272/0 (socket says 172.21.15.176:59272) 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24cf7fe640 1 -- 172.21.15.176:0/3549602787 learned_addr learned my addr 172.21.15.176:0/3549602787 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201441 0 0) 0x7f24d0070da0 con 0x7f24d00b1780 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b8003520 con 0x7f24d00b1780 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010973676 0 0) 0x7f24d009adb0 con 0x7f24d013d7e0 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:39:15.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d0070da0 con 0x7f24d013d7e0 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651254061 0 0) 0x7f24d0070b30 con 0x7f24d0143be0 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d009adb0 con 0x7f24d0143be0 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1241340114 0 0) 0x7f24b8003520 con 0x7f24d00b1780 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 >> v1:172.21.15.114:6790/0 conn(0x7f24d013d7e0 legacy=0x7f24d01434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 >> v1:172.21.15.114:6789/0 conn(0x7f24d0143be0 legacy=0x7f24d014b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 monclient: found mon.noname-c 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d0144070 con 0x7f24d00b1780 2024-03-29T10:39:15.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24d5c74640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f24c0001690 con 0x7f24d00b1780 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient: dump: 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.276 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24d5c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24c0001870 con 0x7f24d00b1780 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 1 -- 172.21.15.176:0/3549602787 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f24c0001c80 con 0x7f24d00b1780 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24d5c74640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24d5c74640 4 set_mon_vals no callback set 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.284+0000 7f24ce7fc640 10 monclient: dump: 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.277 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.285+0000 7f24d5c74640 10 monclient: shutdown 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.285+0000 7f24d5c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.285+0000 7f24d5c74640 1 -- 172.21.15.176:0/3549602787 >> v1:172.21.15.176:6789/0 conn(0x7f24d00b1780 legacy=0x7f24d013d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.285+0000 7f24d5c74640 1 -- 172.21.15.176:0/3549602787 shutdown_connections 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.285+0000 7f24d5c74640 1 -- 172.21.15.176:0/3549602787 >> 172.21.15.176:0/3549602787 conn(0x7f24d00b58f0 msgr2=0x7f24d00b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.285+0000 7f24d5c74640 1 -- 172.21.15.176:0/3549602787 shutdown_connections 2024-03-29T10:39:15.278 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.285+0000 7f24d5c74640 1 -- 172.21.15.176:0/3549602787 wait complete. 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.285+0000 7f24d5c74640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.285+0000 7f24d5c74640 10 monclient: monmap: 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 1 librados: starting msgr at 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 1 librados: starting objecter 2024-03-29T10:39:15.279 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 1 Processor -- start 2024-03-29T10:39:15.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 1 -- start start 2024-03-29T10:39:15.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 1 librados: setting wanted keys 2024-03-29T10:39:15.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 1 librados: calling monclient init 2024-03-29T10:39:15.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 10 monclient: init 2024-03-29T10:39:15.280 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:15.286+0000 7fdf5662b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 10 monclient(hunting): picked mon.noname-c con 0x7f24d00b1780 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 10 monclient(hunting): picked mon.noname-a con 0x7f24d013d7e0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.286+0000 7f24d5c74640 10 monclient(hunting): picked mon.noname-b con 0x7f24d0143be0 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d0138240 con 0x7f24d013d7e0 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d006c650 con 0x7f24d0143be0 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d006b650 con 0x7f24d00b1780 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 10 monclient(hunting): authenticate will time out at 823.571065s 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24ceffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f24d013d7e0 0x7f24d0132970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.176:50264/0 (socket says 172.21.15.176:50264) 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24cf7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f24d00b1780 0x7f24d01da430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.176:50486/0 (socket says 172.21.15.176:50486) 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24ceffd640 1 -- 172.21.15.176:0/480745055 learned_addr learned my addr 172.21.15.176:0/480745055 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24cffff640 1 --1- 172.21.15.176:0/480745055 >> v1:172.21.15.176:6789/0 conn(0x7f24d0143be0 0x7f24d01e27d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59288/0 (socket says 172.21.15.176:59288) 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 client.?.objecter ms_handle_connect 0x7f24d0143be0 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 client.?.objecter ms_handle_connect 0x7f24d00b1780 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 client.?.objecter ms_handle_connect 0x7f24d013d7e0 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318803718 0 0) 0x7f24d006c650 con 0x7f24d0143be0 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24bc004eb0 con 0x7f24d0143be0 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883129661 0 0) 0x7f24d006b650 con 0x7f24d00b1780 2024-03-29T10:39:15.286 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d006c650 con 0x7f24d00b1780 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890397096 0 0) 0x7f24d0138240 con 0x7f24d013d7e0 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d006b650 con 0x7f24d013d7e0 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3485646467 0 0) 0x7f24bc004eb0 con 0x7f24d0143be0 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24d0138240 con 0x7f24d0143be0 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f24c8001bd0 con 0x7f24d0143be0 2024-03-29T10:39:15.287 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): dump: 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2816698784 0 0) 0x7f24d006c650 con 0x7f24d00b1780 2024-03-29T10:39:15.288 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24bc004eb0 con 0x7f24d00b1780 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f24c00019e0 con 0x7f24d00b1780 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): dump: 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.289 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4251304403 0 0) 0x7f24d006b650 con 0x7f24d013d7e0 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24d006c650 con 0x7f24d013d7e0 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f24c40018b0 con 0x7f24d013d7e0 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient(hunting): dump: 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.290 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2662511756 0 0) 0x7f24d0138240 con 0x7f24d0143be0 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 >> v1:172.21.15.114:6790/0 conn(0x7f24d00b1780 legacy=0x7f24d01da430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 >> v1:172.21.15.114:6789/0 conn(0x7f24d013d7e0 legacy=0x7f24d0132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 monclient: found mon.b 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d006b7e0 con 0x7f24d0143be0 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 10 monclient: _renew_subs 2024-03-29T10:39:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 1 -- 172.21.15.176:0/480745055 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d01dab40 con 0x7f24d0143be0 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 10 monclient: _renew_subs 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 1 -- 172.21.15.176:0/480745055 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d00af540 con 0x7f24d0143be0 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24d5c74640 1 librados: init done 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24c8003990 con 0x7f24d0143be0 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f24c8003d20 con 0x7f24d0143be0 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24cdffb640 4 set_mon_vals no callback set 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.292 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.287+0000 7f24b7fff640 10 monclient: dump: 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.288+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f24c80042d0 con 0x7f24d0143be0 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.288+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2262797839 0 0) 0x7f24c80069c0 con 0x7f24d0143be0 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.288+0000 7f24b7fff640 10 client.4135.objecter ms_dispatch 0x7f24d0000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.288+0000 7f24b7fff640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:39:15.293 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.288+0000 7f24b7fff640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:39:15.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.288+0000 7f24b7fff640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T10:39:15.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.288+0000 7f24d5c74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.288+0000 7f24d5c74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.288+0000 7f24d5c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.288+0000 7f24d5c74640 1 -- 172.21.15.176:0/480745055 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b0005080 con 0x7f24d0143be0 2024-03-29T10:39:15.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.290+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f24c8006530 con 0x7f24d0143be0 2024-03-29T10:39:15.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.290+0000 7f24b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.294 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.290+0000 7f24b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:15.299 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:15.315+0000 7fbb4763e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:39:15.315 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:15.331+0000 7fdf5662b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:39:15.347 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.363+0000 7f24d5c74640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d", "id": 3}] 2024-03-29T10:39:15.347 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.363+0000 7f24d5c74640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d", "id": 3}] 2024-03-29T10:39:15.347 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.363+0000 7f24d5c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.347 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.363+0000 7f24d5c74640 1 -- 172.21.15.176:0/480745055 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d", "id": 3} v 0) v1 -- 0x7f24b00058b0 con 0x7f24d0143be0 2024-03-29T10:39:15.355 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.371+0000 7f24b7fff640 1 -- 172.21.15.176:0/480745055 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 256614449 0 525046306) 0x7f24c8006810 con 0x7f24d0143be0 2024-03-29T10:39:15.355 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.371+0000 7f24b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d", "id": 3}] 2024-03-29T10:39:15.355 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.371+0000 7f24b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:15.355 INFO:teuthology.orchestra.run.smithi176.stdout:3 2024-03-29T10:39:15.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.371+0000 7f24d5c74640 10 librados: watch_flush enter 2024-03-29T10:39:15.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.371+0000 7f24d5c74640 10 librados: watch_flush exit 2024-03-29T10:39:15.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.371+0000 7f24d5c74640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T10:39:15.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.371+0000 7f24d5c74640 10 client.4135.objecter successfully canceled tick 2024-03-29T10:39:15.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.372+0000 7f24d5c74640 10 monclient: shutdown 2024-03-29T10:39:15.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.372+0000 7f24d5c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.372+0000 7f24d5c74640 1 -- 172.21.15.176:0/480745055 >> v1:172.21.15.176:6789/0 conn(0x7f24d0143be0 legacy=0x7f24d01e27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.372+0000 7f24d5c74640 1 -- 172.21.15.176:0/480745055 shutdown_connections 2024-03-29T10:39:15.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.372+0000 7f24d5c74640 1 -- 172.21.15.176:0/480745055 >> 172.21.15.176:0/480745055 conn(0x7f24d00b58f0 msgr2=0x7f24d0069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.372+0000 7f24d5c74640 1 -- 172.21.15.176:0/480745055 shutdown_connections 2024-03-29T10:39:15.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.372+0000 7f24d5c74640 1 -- 172.21.15.176:0/480745055 wait complete. 2024-03-29T10:39:15.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.372+0000 7f24d5c74640 1 librados: shutdown 2024-03-29T10:39:15.366 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 1585ada0-eaef-41fc-b41d-179f7c1756b3 4 2024-03-29T10:39:15.415 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:15.431+0000 7fbb4763e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:39:15.436 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:15.452+0000 7fdf5662b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:39:15.465 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.481+0000 7f23c7554640 10 monclient: get_monmap_and_config 2024-03-29T10:39:15.465 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.481+0000 7f23c7554640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.465 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.481+0000 7f23c7554640 10 monclient: monmap: 2024-03-29T10:39:15.466 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:15.466 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.466 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:39:15.482646+0000 2024-03-29T10:39:15.466 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:39:15.482646+0000 2024-03-29T10:39:15.466 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.466 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.466 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:15.466 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:15.466 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:15.466 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.482+0000 7f23c7554640 1 Processor -- start 2024-03-29T10:39:15.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.482+0000 7f23c7554640 1 -- start start 2024-03-29T10:39:15.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.482+0000 7f23c7554640 10 monclient: init 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 10 monclient(hunting): picked mon.noname-b con 0x7f23c009ac40 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 10 monclient(hunting): picked mon.noname-a con 0x7f23c009c870 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 10 monclient(hunting): picked mon.noname-c con 0x7f23c00b4360 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c009dec0 con 0x7f23c009c870 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c009da30 con 0x7f23c00b4360 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c00b39c0 con 0x7f23c009ac40 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 10 monclient(hunting): authenticate will time out at 823.767202s 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c5af1640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f23c00b4360 0x7f23c014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59300/0 (socket says 172.21.15.176:59300) 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c5af1640 1 -- 172.21.15.176:0/2251647357 learned_addr learned my addr 172.21.15.176:0/2251647357 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982320331 0 0) 0x7f23c009da30 con 0x7f23c00b4360 2024-03-29T10:39:15.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ac003520 con 0x7f23c00b4360 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356153373 0 0) 0x7f23c00b39c0 con 0x7f23c009ac40 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c009da30 con 0x7f23c009ac40 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824285057 0 0) 0x7f23b0001040 con 0x7f23c009c870 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c00b39c0 con 0x7f23c009c870 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2596515942 0 0) 0x7f23ac003520 con 0x7f23c00b4360 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 >> v1:172.21.15.114:6790/0 conn(0x7f23c009ac40 legacy=0x7f23c00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 >> v1:172.21.15.114:6789/0 conn(0x7f23c009c870 legacy=0x7f23c013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 monclient: found mon.noname-c 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c00b5160 con 0x7f23c00b4360 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f23b8001a10 con 0x7f23c00b4360 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient: dump: 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.480 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b8001bf0 con 0x7f23c00b4360 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 1 -- 172.21.15.176:0/2251647357 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f23b8002000 con 0x7f23c00b4360 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23c7554640 4 set_mon_vals no callback set 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.483+0000 7f23b7fff640 10 monclient: dump: 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.481 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.484+0000 7f23c7554640 10 monclient: shutdown 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.484+0000 7f23c7554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.484+0000 7f23c7554640 1 -- 172.21.15.176:0/2251647357 >> v1:172.21.15.176:6789/0 conn(0x7f23c00b4360 legacy=0x7f23c014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.484+0000 7f23c7554640 1 -- 172.21.15.176:0/2251647357 shutdown_connections 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.484+0000 7f23c7554640 1 -- 172.21.15.176:0/2251647357 >> 172.21.15.176:0/2251647357 conn(0x7f23c0066b00 msgr2=0x7f23c0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.484+0000 7f23c7554640 1 -- 172.21.15.176:0/2251647357 shutdown_connections 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.484+0000 7f23c7554640 1 -- 172.21.15.176:0/2251647357 wait complete. 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.485+0000 7f23c7554640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.485+0000 7f23c7554640 10 monclient: monmap: 2024-03-29T10:39:15.482 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.485+0000 7f23c7554640 1 librados: starting msgr at 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.485+0000 7f23c7554640 1 librados: starting objecter 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.485+0000 7f23c7554640 1 Processor -- start 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.485+0000 7f23c7554640 1 -- start start 2024-03-29T10:39:15.483 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.485+0000 7f23c7554640 1 librados: setting wanted keys 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.485+0000 7f23c7554640 1 librados: calling monclient init 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.485+0000 7f23c7554640 10 monclient: init 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 10 monclient(hunting): picked mon.noname-b con 0x7f23c009ac40 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 10 monclient(hunting): picked mon.noname-c con 0x7f23c00b7310 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 10 monclient(hunting): picked mon.noname-a con 0x7f23c01d7de0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c00b78c0 con 0x7f23c01d7de0 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c00b7d50 con 0x7f23c009ac40 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c0146480 con 0x7f23c00b7310 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 10 monclient(hunting): authenticate will time out at 823.770222s 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c52f0640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f23c009ac40 0x7f23c00b6a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59310/0 (socket says 172.21.15.176:59310) 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c52f0640 1 -- 172.21.15.176:0/2982348075 learned_addr learned my addr 172.21.15.176:0/2982348075 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:15.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 client.?.objecter ms_handle_connect 0x7f23c009ac40 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 client.?.objecter ms_handle_connect 0x7f23c01d7de0 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160081943 0 0) 0x7f23c00b7d50 con 0x7f23c009ac40 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a4004eb0 con 0x7f23c009ac40 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 client.?.objecter ms_handle_connect 0x7f23c00b7310 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668890983 0 0) 0x7f23c00b78c0 con 0x7f23c01d7de0 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c00b7d50 con 0x7f23c01d7de0 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128117912 0 0) 0x7f23c0146480 con 0x7f23c00b7310 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c00b78c0 con 0x7f23c00b7310 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 746967566 0 0) 0x7f23a4004eb0 con 0x7f23c009ac40 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23c0146480 con 0x7f23c009ac40 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f23bc001a30 con 0x7f23c009ac40 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4079026033 0 0) 0x7f23c00b7d50 con 0x7f23c01d7de0 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23a4004eb0 con 0x7f23c01d7de0 2024-03-29T10:39:15.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f23b8001f00 con 0x7f23c01d7de0 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3116916080 0 0) 0x7f23c00b78c0 con 0x7f23c00b7310 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23c00b7d50 con 0x7f23c00b7310 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f23b00016d0 con 0x7f23c00b7310 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.489 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1159177423 0 0) 0x7f23c0146480 con 0x7f23c009ac40 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 >> v1:172.21.15.114:6790/0 conn(0x7f23c00b7310 legacy=0x7f23c01d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 >> v1:172.21.15.114:6789/0 conn(0x7f23c01d7de0 legacy=0x7f23c01e2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 monclient: found mon.b 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c01e68d0 con 0x7f23c009ac40 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 10 monclient: _renew_subs 2024-03-29T10:39:15.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.486+0000 7f23c7554640 1 -- 172.21.15.176:0/2982348075 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c01e38a0 con 0x7f23c009ac40 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23bc001cf0 con 0x7f23c009ac40 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f23bc003a60 con 0x7f23c009ac40 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 10 monclient: dump: 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.491 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f23bc003f50 con 0x7f23c009ac40 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b77fe640 4 set_mon_vals no callback set 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23c7554640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23c7554640 10 monclient: _renew_subs 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23c7554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23c7554640 1 -- 172.21.15.176:0/2982348075 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c01e3d80 con 0x7f23c009ac40 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23c7554640 1 librados: init done 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2635767000 0 0) 0x7f23bc004430 con 0x7f23c009ac40 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 10 client.4141.objecter ms_dispatch 0x7f23c0000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.487+0000 7f23b5ffb640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T10:39:15.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.488+0000 7f23c7554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.493 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.488+0000 7f23c7554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.493 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.488+0000 7f23c7554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.493 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.488+0000 7f23c7554640 1 -- 172.21.15.176:0/2982348075 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c0143470 con 0x7f23c009ac40 2024-03-29T10:39:15.493 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.489+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f23bc005be0 con 0x7f23c009ac40 2024-03-29T10:39:15.493 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.490+0000 7f23b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.493 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.490+0000 7f23b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:15.493 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:15.501+0000 7fbb4763e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:39:15.509 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:15.526+0000 7fdf5662b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:39:15.531 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:15.547+0000 7fbb4763e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:39:15.546 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.562+0000 7f23c7554640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1585ada0-eaef-41fc-b41d-179f7c1756b3", "id": 4}] 2024-03-29T10:39:15.546 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.562+0000 7f23c7554640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1585ada0-eaef-41fc-b41d-179f7c1756b3", "id": 4}] 2024-03-29T10:39:15.546 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.562+0000 7f23c7554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.547 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.562+0000 7f23c7554640 1 -- 172.21.15.176:0/2982348075 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1585ada0-eaef-41fc-b41d-179f7c1756b3", "id": 4} v 0) v1 -- 0x7f23c009b020 con 0x7f23c009ac40 2024-03-29T10:39:15.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.570+0000 7f23b5ffb640 1 -- 172.21.15.176:0/2982348075 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1585ada0-eaef-41fc-b41d-179f7c1756b3", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3121850637 0 1697012583) 0x7f23bc0057c0 con 0x7f23c009ac40 2024-03-29T10:39:15.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.570+0000 7f23b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1585ada0-eaef-41fc-b41d-179f7c1756b3", "id": 4}] 2024-03-29T10:39:15.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.570+0000 7f23b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:15.555 INFO:teuthology.orchestra.run.smithi176.stdout:4 2024-03-29T10:39:15.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 10 librados: watch_flush enter 2024-03-29T10:39:15.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 10 librados: watch_flush exit 2024-03-29T10:39:15.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T10:39:15.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 10 client.4141.objecter successfully canceled tick 2024-03-29T10:39:15.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 10 monclient: shutdown 2024-03-29T10:39:15.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 1 -- 172.21.15.176:0/2982348075 >> v1:172.21.15.176:6789/0 conn(0x7f23c009ac40 legacy=0x7f23c00b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 1 -- 172.21.15.176:0/2982348075 shutdown_connections 2024-03-29T10:39:15.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 1 -- 172.21.15.176:0/2982348075 >> 172.21.15.176:0/2982348075 conn(0x7f23c0066b00 msgr2=0x7f23c013a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 1 -- 172.21.15.176:0/2982348075 shutdown_connections 2024-03-29T10:39:15.557 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 1 -- 172.21.15.176:0/2982348075 wait complete. 2024-03-29T10:39:15.557 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.571+0000 7f23c7554640 1 librados: shutdown 2024-03-29T10:39:15.557 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:15.573+0000 7fdf5662b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:39:15.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4 5 2024-03-29T10:39:15.584 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:15.600+0000 7fbb4763e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:39:15.614 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:15.630+0000 7fdf5662b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:39:15.631 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:15.647+0000 7fbb4763e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:39:15.664 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:15.680+0000 7fdf5662b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:39:15.667 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.683+0000 7f0dc40c1640 10 monclient: get_monmap_and_config 2024-03-29T10:39:15.667 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.683+0000 7f0dc40c1640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.667 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.683+0000 7f0dc40c1640 10 monclient: monmap: 2024-03-29T10:39:15.667 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:15.667 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.667 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:39:15.684209+0000 2024-03-29T10:39:15.667 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:39:15.684209+0000 2024-03-29T10:39:15.667 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.667 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.667 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:15.667 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:15.668 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:15.668 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.676 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.683+0000 7f0dc40c1640 1 Processor -- start 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 1 -- start start 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 10 monclient: init 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 10 monclient(hunting): picked mon.noname-b con 0x7f0dbc09ac40 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 10 monclient(hunting): picked mon.noname-a con 0x7f0dbc09c870 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 10 monclient(hunting): picked mon.noname-c con 0x7f0dbc0b4360 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc09dec0 con 0x7f0dbc09c870 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc09da30 con 0x7f0dbc0b4360 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc0b39c0 con 0x7f0dbc09ac40 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc40c1640 10 monclient(hunting): authenticate will time out at 823.968622s 2024-03-29T10:39:15.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc265e640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f0dbc0b4360 0x7f0dbc14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59316/0 (socket says 172.21.15.176:59316) 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc265e640 1 -- 172.21.15.176:0/1883768670 learned_addr learned my addr 172.21.15.176:0/1883768670 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729769518 0 0) 0x7f0dbc09da30 con 0x7f0dbc0b4360 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc0e5b640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc0e5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc0e5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc0e5b640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.684+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da8003520 con 0x7f0dbc0b4360 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207843722 0 0) 0x7f0dac001110 con 0x7f0dbc09c870 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:39:15.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc09da30 con 0x7f0dbc09c870 2024-03-29T10:39:15.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257630593 0 0) 0x7f0dbc0b39c0 con 0x7f0dbc09ac40 2024-03-29T10:39:15.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:39:15.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dac001110 con 0x7f0dbc09ac40 2024-03-29T10:39:15.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 521095544 0 0) 0x7f0da8003520 con 0x7f0dbc0b4360 2024-03-29T10:39:15.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 >> v1:172.21.15.114:6790/0 conn(0x7f0dbc09ac40 legacy=0x7f0dbc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 >> v1:172.21.15.114:6789/0 conn(0x7f0dbc09c870 legacy=0x7f0dbc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 monclient: found mon.noname-c 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc0b5160 con 0x7f0dbc0b4360 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0db4001a10 con 0x7f0dbc0b4360 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc40c1640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc40c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient: dump: 2024-03-29T10:39:15.680 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc40c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0db4001bf0 con 0x7f0dbc0b4360 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 1 -- 172.21.15.176:0/1883768670 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0db4002000 con 0x7f0dbc0b4360 2024-03-29T10:39:15.681 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc40c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc40c1640 4 set_mon_vals no callback set 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc0e5b640 10 monclient: dump: 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.682 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc40c1640 10 monclient: shutdown 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc40c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc40c1640 1 -- 172.21.15.176:0/1883768670 >> v1:172.21.15.176:6789/0 conn(0x7f0dbc0b4360 legacy=0x7f0dbc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc40c1640 1 -- 172.21.15.176:0/1883768670 shutdown_connections 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc40c1640 1 -- 172.21.15.176:0/1883768670 >> 172.21.15.176:0/1883768670 conn(0x7f0dbc066b00 msgr2=0x7f0dbc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.685+0000 7f0dc40c1640 1 -- 172.21.15.176:0/1883768670 shutdown_connections 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.686+0000 7f0dc40c1640 1 -- 172.21.15.176:0/1883768670 wait complete. 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.686+0000 7f0dc40c1640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.686+0000 7f0dc40c1640 10 monclient: monmap: 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T10:39:15.683 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.686+0000 7f0dc40c1640 1 librados: starting msgr at 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.686+0000 7f0dc40c1640 1 librados: starting objecter 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.686+0000 7f0dc40c1640 1 Processor -- start 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.686+0000 7f0dc40c1640 1 -- start start 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.686+0000 7f0dc40c1640 1 librados: setting wanted keys 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.686+0000 7f0dc40c1640 1 librados: calling monclient init 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.686+0000 7f0dc40c1640 10 monclient: init 2024-03-29T10:39:15.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc40c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc40c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc40c1640 10 monclient(hunting): picked mon.noname-a con 0x7f0dbc09ac40 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc40c1640 10 monclient(hunting): picked mon.noname-b con 0x7f0dbc145010 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc40c1640 10 monclient(hunting): picked mon.noname-c con 0x7f0dbc145690 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc40c1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc145cb0 con 0x7f0dbc09ac40 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc40c1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc1389f0 con 0x7f0dbc145010 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc40c1640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc0ad7f0 con 0x7f0dbc145690 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc40c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc40c1640 10 monclient(hunting): authenticate will time out at 823.971502s 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc165c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f0dbc145010 0x7f0dbc137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59330/0 (socket says 172.21.15.176:59330) 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0dc165c640 1 -- 172.21.15.176:0/3434969216 learned_addr learned my addr 172.21.15.176:0/3434969216 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 client.?.objecter ms_handle_connect 0x7f0dbc145010 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 client.?.objecter ms_handle_connect 0x7f0dbc145690 2024-03-29T10:39:15.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 client.?.objecter ms_handle_connect 0x7f0dbc09ac40 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990701552 0 0) 0x7f0dbc1389f0 con 0x7f0dbc145010 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da0004eb0 con 0x7f0dbc145010 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584093925 0 0) 0x7f0dbc0ad7f0 con 0x7f0dbc145690 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.686 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc1389f0 con 0x7f0dbc145690 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886272858 0 0) 0x7f0dbc145cb0 con 0x7f0dbc09ac40 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.687+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dbc0ad7f0 con 0x7f0dbc09ac40 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1146801478 0 0) 0x7f0da0004eb0 con 0x7f0dbc145010 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dbc145cb0 con 0x7f0dbc145010 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0dac0019a0 con 0x7f0dbc145010 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient(hunting): dump: 2024-03-29T10:39:15.687 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2350057549 0 0) 0x7f0dbc1389f0 con 0x7f0dbc145690 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0da0004eb0 con 0x7f0dbc145690 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0db4001f00 con 0x7f0dbc145690 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.688 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient(hunting): dump: 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545101552 0 0) 0x7f0dbc0ad7f0 con 0x7f0dbc09ac40 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dbc1389f0 con 0x7f0dbc09ac40 2024-03-29T10:39:15.689 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0db8001830 con 0x7f0dbc09ac40 2024-03-29T10:39:15.690 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.690 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:15.690 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient(hunting): dump: 2024-03-29T10:39:15.690 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.690 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.690 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.690 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.690 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.690 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.690 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1683014831 0 0) 0x7f0dbc145cb0 con 0x7f0dbc145010 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 >> v1:172.21.15.114:6790/0 conn(0x7f0dbc145690 legacy=0x7f0dbc1382e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 >> v1:172.21.15.114:6789/0 conn(0x7f0dbc09ac40 legacy=0x7f0dbc1d7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 monclient: found mon.b 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc1424e0 con 0x7f0dbc145010 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.691 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0dc40c1640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0dc40c1640 10 monclient: _renew_subs 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0dc40c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0dc40c1640 1 -- 172.21.15.176:0/3434969216 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dbc142790 con 0x7f0dbc145010 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0dc40c1640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0dc40c1640 10 monclient: _renew_subs 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0dc40c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0dc40c1640 1 -- 172.21.15.176:0/3434969216 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc1d7a60 con 0x7f0dbc145010 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0dc40c1640 1 librados: init done 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dac003610 con 0x7f0dbc145010 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0dac003920 con 0x7f0dbc145010 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db3fff640 4 set_mon_vals no callback set 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 monclient: dump: 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0dac003ed0 con 0x7f0dbc145010 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3143861214 0 0) 0x7f0dac0067d0 con 0x7f0dbc145010 2024-03-29T10:39:15.693 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 10 client.4147.objecter ms_dispatch 0x7f0dbc000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:39:15.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:39:15.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:39:15.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.688+0000 7f0db27fc640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T10:39:15.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.689+0000 7f0dc40c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.689+0000 7f0dc40c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.689+0000 7f0dc40c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.689+0000 7f0dc40c1640 1 -- 172.21.15.176:0/3434969216 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da4005080 con 0x7f0dbc145010 2024-03-29T10:39:15.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.692+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0dac0042c0 con 0x7f0dbc145010 2024-03-29T10:39:15.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.692+0000 7f0db27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.692+0000 7f0db27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:15.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.765+0000 7f0dc40c1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4", "id": 5}] 2024-03-29T10:39:15.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.765+0000 7f0dc40c1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4", "id": 5}] 2024-03-29T10:39:15.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.765+0000 7f0dc40c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.765+0000 7f0dc40c1640 1 -- 172.21.15.176:0/3434969216 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4", "id": 5} v 0) v1 -- 0x7f0da40058b0 con 0x7f0dbc145010 2024-03-29T10:39:15.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.770+0000 7f0db27fc640 1 -- 172.21.15.176:0/3434969216 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3865511150 0 1988412176) 0x7f0dac006250 con 0x7f0dbc145010 2024-03-29T10:39:15.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.770+0000 7f0db27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4", "id": 5}] 2024-03-29T10:39:15.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.770+0000 7f0db27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:15.754 INFO:teuthology.orchestra.run.smithi176.stdout:5 2024-03-29T10:39:15.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.770+0000 7f0dc40c1640 10 librados: watch_flush enter 2024-03-29T10:39:15.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.770+0000 7f0dc40c1640 10 librados: watch_flush exit 2024-03-29T10:39:15.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.771+0000 7f0dc40c1640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T10:39:15.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.771+0000 7f0dc40c1640 10 client.4147.objecter successfully canceled tick 2024-03-29T10:39:15.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.771+0000 7f0dc40c1640 10 monclient: shutdown 2024-03-29T10:39:15.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.771+0000 7f0dc40c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.771+0000 7f0dc40c1640 1 -- 172.21.15.176:0/3434969216 >> v1:172.21.15.176:6789/0 conn(0x7f0dbc145010 legacy=0x7f0dbc137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.771+0000 7f0dc40c1640 1 -- 172.21.15.176:0/3434969216 shutdown_connections 2024-03-29T10:39:15.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.771+0000 7f0dc40c1640 1 -- 172.21.15.176:0/3434969216 >> 172.21.15.176:0/3434969216 conn(0x7f0dbc066b00 msgr2=0x7f0dbc13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.771+0000 7f0dc40c1640 1 -- 172.21.15.176:0/3434969216 shutdown_connections 2024-03-29T10:39:15.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.771+0000 7f0dc40c1640 1 -- 172.21.15.176:0/3434969216 wait complete. 2024-03-29T10:39:15.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.771+0000 7f0dc40c1640 1 librados: shutdown 2024-03-29T10:39:15.765 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 9230491b-7e8a-4789-8f62-c189a38c9ef3 6 2024-03-29T10:39:15.864 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.880+0000 7fe046d4b640 10 monclient: get_monmap_and_config 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.880+0000 7fe046d4b640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.880+0000 7fe046d4b640 10 monclient: monmap: 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:39:15.881666+0000 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:39:15.881666+0000 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:15.865 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.881+0000 7fe046d4b640 1 Processor -- start 2024-03-29T10:39:15.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.881+0000 7fe046d4b640 1 -- start start 2024-03-29T10:39:15.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.881+0000 7fe046d4b640 10 monclient: init 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.881+0000 7fe046d4b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.881+0000 7fe046d4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.881+0000 7fe046d4b640 10 monclient(hunting): picked mon.noname-a con 0x7fe04009ac40 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fe04009c870 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 10 monclient(hunting): picked mon.noname-b con 0x7fe0400b4360 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04009dec0 con 0x7fe04009ac40 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04009da30 con 0x7fe04009c870 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0400b39c0 con 0x7fe0400b4360 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 10 monclient(hunting): authenticate will time out at 824.166085s 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03ffff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe04009c870 0x7fe04013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59332/0 (socket says 172.21.15.176:59332) 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03ffff640 1 -- 172.21.15.176:0/2428540125 learned_addr learned my addr 172.21.15.176:0/2428540125 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361534061 0 0) 0x7fe04009da30 con 0x7fe04009c870 2024-03-29T10:39:15.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe030003520 con 0x7fe04009c870 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646729874 0 0) 0x7fe034001110 con 0x7fe04009ac40 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe04009da30 con 0x7fe04009ac40 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214758851 0 0) 0x7fe0400b39c0 con 0x7fe0400b4360 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe034001110 con 0x7fe0400b4360 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4043245741 0 0) 0x7fe030003520 con 0x7fe04009c870 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 >> v1:172.21.15.114:6790/0 conn(0x7fe0400b4360 legacy=0x7fe04014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 >> v1:172.21.15.114:6789/0 conn(0x7fe04009ac40 legacy=0x7fe0400b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 monclient: found mon.noname-c 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0400b5160 con 0x7fe04009c870 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:39:15.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe02c001690 con 0x7fe04009c870 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient: dump: 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe02c001870 con 0x7fe04009c870 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 1 -- 172.21.15.176:0/2428540125 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe02c001c80 con 0x7fe04009c870 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe046d4b640 4 set_mon_vals no callback set 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.882+0000 7fe03f7fe640 10 monclient: dump: 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.879 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.883+0000 7fe046d4b640 10 monclient: shutdown 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.883+0000 7fe046d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.883+0000 7fe046d4b640 1 -- 172.21.15.176:0/2428540125 >> v1:172.21.15.176:6789/0 conn(0x7fe04009c870 legacy=0x7fe04013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.883+0000 7fe046d4b640 1 -- 172.21.15.176:0/2428540125 shutdown_connections 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.883+0000 7fe046d4b640 1 -- 172.21.15.176:0/2428540125 >> 172.21.15.176:0/2428540125 conn(0x7fe040066b00 msgr2=0x7fe040066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.883+0000 7fe046d4b640 1 -- 172.21.15.176:0/2428540125 shutdown_connections 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.883+0000 7fe046d4b640 1 -- 172.21.15.176:0/2428540125 wait complete. 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.883+0000 7fe046d4b640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.883+0000 7fe046d4b640 10 monclient: monmap: 2024-03-29T10:39:15.880 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.884+0000 7fe046d4b640 1 librados: starting msgr at 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.884+0000 7fe046d4b640 1 librados: starting objecter 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.884+0000 7fe046d4b640 1 Processor -- start 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.884+0000 7fe046d4b640 1 -- start start 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.884+0000 7fe046d4b640 1 librados: setting wanted keys 2024-03-29T10:39:15.881 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.884+0000 7fe046d4b640 1 librados: calling monclient init 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.884+0000 7fe046d4b640 10 monclient: init 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.884+0000 7fe046d4b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 10 monclient(hunting): picked mon.noname-b con 0x7fe04009ac40 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fe0401d7bc0 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 10 monclient(hunting): picked mon.noname-a con 0x7fe0400b17f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04013b890 con 0x7fe0400b17f0 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0401d76a0 con 0x7fe04009ac40 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0401406b0 con 0x7fe0401d7bc0 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 10 monclient(hunting): authenticate will time out at 824.169087s 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe044ae7640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe04009ac40 0x7fe04013ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59336/0 (socket says 172.21.15.176:59336) 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe044ae7640 1 -- 172.21.15.176:0/2320483516 learned_addr learned my addr 172.21.15.176:0/2320483516 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:15.882 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 client.?.objecter ms_handle_connect 0x7fe04009ac40 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 client.?.objecter ms_handle_connect 0x7fe0401d7bc0 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 client.?.objecter ms_handle_connect 0x7fe0400b17f0 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497474387 0 0) 0x7fe0401d76a0 con 0x7fe04009ac40 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe024004eb0 con 0x7fe04009ac40 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511287402 0 0) 0x7fe0401406b0 con 0x7fe0401d7bc0 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.883 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0401d76a0 con 0x7fe0401d7bc0 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229994280 0 0) 0x7fe04013b890 con 0x7fe0400b17f0 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0401406b0 con 0x7fe0400b17f0 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3272468942 0 0) 0x7fe024004eb0 con 0x7fe04009ac40 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe04013b890 con 0x7fe04009ac40 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe0340019a0 con 0x7fe04009ac40 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:15.884 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): dump: 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3390269313 0 0) 0x7fe0401d76a0 con 0x7fe0401d7bc0 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe024004eb0 con 0x7fe0401d7bc0 2024-03-29T10:39:15.885 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe02c001b80 con 0x7fe0401d7bc0 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): dump: 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.886 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3303010421 0 0) 0x7fe0401406b0 con 0x7fe0400b17f0 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0401d76a0 con 0x7fe0400b17f0 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe038001b50 con 0x7fe0400b17f0 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient(hunting): dump: 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.887 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3475706206 0 0) 0x7fe04013b890 con 0x7fe04009ac40 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 >> v1:172.21.15.114:6790/0 conn(0x7fe0401d7bc0 legacy=0x7fe04013aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 >> v1:172.21.15.114:6789/0 conn(0x7fe0400b17f0 legacy=0x7fe04013b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 monclient: found mon.b 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe040140840 con 0x7fe04009ac40 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:39:15.888 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 10 monclient: _renew_subs 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 1 -- 172.21.15.176:0/2320483516 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0400b6f40 con 0x7fe04009ac40 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe046d4b640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.885+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe034001c40 con 0x7fe04009ac40 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe046d4b640 10 monclient: _renew_subs 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe046d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe046d4b640 1 -- 172.21.15.176:0/2320483516 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0401d6f40 con 0x7fe04009ac40 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe046d4b640 1 librados: init done 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe034003a60 con 0x7fe04009ac40 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03d7fa640 10 monclient: dump: 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:15.889 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:15.890 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.890 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:15.890 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.890 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:15.890 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:15.890 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:15.890 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:15.890 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:15.890 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03effd640 4 set_mon_vals no callback set 2024-03-29T10:39:15.890 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe034003fd0 con 0x7fe04009ac40 2024-03-29T10:39:15.890 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 278224859 0 0) 0x7fe034005db0 con 0x7fe04009ac40 2024-03-29T10:39:15.891 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03d7fa640 10 client.4153.objecter ms_dispatch 0x7fe040000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:39:15.891 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03d7fa640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:39:15.891 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03d7fa640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:39:15.891 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.886+0000 7fe03d7fa640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T10:39:15.891 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.887+0000 7fe046d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.891 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.887+0000 7fe046d4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.891 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.887+0000 7fe046d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.891 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.887+0000 7fe046d4b640 1 -- 172.21.15.176:0/2320483516 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0401d7830 con 0x7fe04009ac40 2024-03-29T10:39:15.891 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.889+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe034004390 con 0x7fe04009ac40 2024-03-29T10:39:15.891 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.889+0000 7fe03d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.891 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.889+0000 7fe03d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:15.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.961+0000 7fe046d4b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9230491b-7e8a-4789-8f62-c189a38c9ef3", "id": 6}] 2024-03-29T10:39:15.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.961+0000 7fe046d4b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9230491b-7e8a-4789-8f62-c189a38c9ef3", "id": 6}] 2024-03-29T10:39:15.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.961+0000 7fe046d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:15.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.961+0000 7fe046d4b640 1 -- 172.21.15.176:0/2320483516 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9230491b-7e8a-4789-8f62-c189a38c9ef3", "id": 6} v 0) v1 -- 0x7fe04009b020 con 0x7fe04009ac40 2024-03-29T10:39:15.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.965+0000 7fe03d7fa640 1 -- 172.21.15.176:0/2320483516 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9230491b-7e8a-4789-8f62-c189a38c9ef3", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 4208625615 0 1113811849) 0x7fe0340057c0 con 0x7fe04009ac40 2024-03-29T10:39:15.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.965+0000 7fe03d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9230491b-7e8a-4789-8f62-c189a38c9ef3", "id": 6}] 2024-03-29T10:39:15.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.965+0000 7fe03d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:15.949 INFO:teuthology.orchestra.run.smithi176.stdout:6 2024-03-29T10:39:15.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 10 librados: watch_flush enter 2024-03-29T10:39:15.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 10 librados: watch_flush exit 2024-03-29T10:39:15.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T10:39:15.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 10 client.4153.objecter successfully canceled tick 2024-03-29T10:39:15.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 10 monclient: shutdown 2024-03-29T10:39:15.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 1 -- 172.21.15.176:0/2320483516 >> v1:172.21.15.176:6789/0 conn(0x7fe04009ac40 legacy=0x7fe04013ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 1 -- 172.21.15.176:0/2320483516 shutdown_connections 2024-03-29T10:39:15.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 1 -- 172.21.15.176:0/2320483516 >> 172.21.15.176:0/2320483516 conn(0x7fe040066b00 msgr2=0x7fe04013d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 1 -- 172.21.15.176:0/2320483516 shutdown_connections 2024-03-29T10:39:15.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 1 -- 172.21.15.176:0/2320483516 wait complete. 2024-03-29T10:39:15.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:15.966+0000 7fe046d4b640 1 librados: shutdown 2024-03-29T10:39:15.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 1164e467-12f7-40e9-9770-7e27c30cfa84 7 2024-03-29T10:39:16.016 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:16.032+0000 7fbb4763e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:39:16.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.077+0000 7feaf7360640 10 monclient: get_monmap_and_config 2024-03-29T10:39:16.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.077+0000 7feaf7360640 10 monclient: build_initial_monmap 2024-03-29T10:39:16.061 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.077+0000 7feaf7360640 10 monclient: monmap: 2024-03-29T10:39:16.061 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:16.061 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.061 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:39:16.078248+0000 2024-03-29T10:39:16.061 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:39:16.078248+0000 2024-03-29T10:39:16.061 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:16.062 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:16.062 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:16.062 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:16.062 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:16.062 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:16.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.077+0000 7feaf7360640 1 Processor -- start 2024-03-29T10:39:16.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 1 -- start start 2024-03-29T10:39:16.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 10 monclient: init 2024-03-29T10:39:16.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:16.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:16.072 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 10 monclient(hunting): picked mon.noname-c con 0x7feaf009ac40 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 10 monclient(hunting): picked mon.noname-a con 0x7feaf009c870 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 10 monclient(hunting): picked mon.noname-b con 0x7feaf00b4360 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf009dec0 con 0x7feaf009c870 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf009da30 con 0x7feaf009ac40 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf00b39c0 con 0x7feaf00b4360 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf7360640 10 monclient(hunting): authenticate will time out at 824.362780s 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf50fc640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7feaf009ac40 0x7feaf00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59350/0 (socket says 172.21.15.176:59350) 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.078+0000 7feaf50fc640 1 -- 172.21.15.176:0/521586955 learned_addr learned my addr 172.21.15.176:0/521586955 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768702939 0 0) 0x7feaf009da30 con 0x7feaf009ac40 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.073 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): my global_id is 4156 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feadc003520 con 0x7feaf009ac40 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828184264 0 0) 0x7feaf00b39c0 con 0x7feaf00b4360 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf009da30 con 0x7feaf00b4360 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657085200 0 0) 0x7feae0001110 con 0x7feaf009c870 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:39:16.074 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf00b39c0 con 0x7feaf009c870 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1205782797 0 0) 0x7feadc003520 con 0x7feaf009ac40 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 >> v1:172.21.15.114:6790/0 conn(0x7feaf00b4360 legacy=0x7feaf014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 >> v1:172.21.15.114:6789/0 conn(0x7feaf009c870 legacy=0x7feaf013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 monclient: found mon.noname-c 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf00b5160 con 0x7feaf009ac40 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feaf7360640 5 monclient: authenticate success, global_id 4156 2024-03-29T10:39:16.075 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7feaec001690 con 0x7feaf009ac40 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feaf7360640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient: dump: 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:16.076 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feaf7360640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feaec0017f0 con 0x7feaf009ac40 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feaf7360640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 1 -- 172.21.15.176:0/521586955 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7feaec001c00 con 0x7feaf009ac40 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feaf7360640 4 set_mon_vals no callback set 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.079+0000 7feae7fff640 10 monclient: dump: 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:16.077 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.080+0000 7feaf7360640 10 monclient: shutdown 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.080+0000 7feaf7360640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.080+0000 7feaf7360640 1 -- 172.21.15.176:0/521586955 >> v1:172.21.15.176:6789/0 conn(0x7feaf009ac40 legacy=0x7feaf00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.080+0000 7feaf7360640 1 -- 172.21.15.176:0/521586955 shutdown_connections 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.080+0000 7feaf7360640 1 -- 172.21.15.176:0/521586955 >> 172.21.15.176:0/521586955 conn(0x7feaf0066b00 msgr2=0x7feaf0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.080+0000 7feaf7360640 1 -- 172.21.15.176:0/521586955 shutdown_connections 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.080+0000 7feaf7360640 1 -- 172.21.15.176:0/521586955 wait complete. 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.080+0000 7feaf7360640 10 monclient: build_initial_monmap 2024-03-29T10:39:16.078 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.080+0000 7feaf7360640 10 monclient: monmap: 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 1 librados: starting msgr at 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 1 librados: starting objecter 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 1 Processor -- start 2024-03-29T10:39:16.079 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 1 -- start start 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 1 librados: setting wanted keys 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 1 librados: calling monclient init 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 10 monclient: init 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 10 monclient(hunting): picked mon.noname-a con 0x7feaf009ac40 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 10 monclient(hunting): picked mon.noname-b con 0x7feaf00ad140 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 10 monclient(hunting): picked mon.noname-c con 0x7feaf01d77e0 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf01d7c20 con 0x7feaf009ac40 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf01d7db0 con 0x7feaf00ad140 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf01d7f40 con 0x7feaf01d77e0 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf7360640 10 monclient(hunting): authenticate will time out at 824.365848s 2024-03-29T10:39:16.080 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf48fb640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7feaf00ad140 0x7feaf00afc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:59352/0 (socket says 172.21.15.176:59352) 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feaf48fb640 1 -- 172.21.15.176:0/875045235 learned_addr learned my addr 172.21.15.176:0/875045235 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feae5ffb640 10 client.?.objecter ms_handle_connect 0x7feaf00ad140 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.081+0000 7feae5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 client.?.objecter ms_handle_connect 0x7feaf01d77e0 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 client.?.objecter ms_handle_connect 0x7feaf009ac40 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885931177 0 0) 0x7feaf01d7db0 con 0x7feaf00ad140 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): my global_id is 4159 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead4004eb0 con 0x7feaf00ad140 2024-03-29T10:39:16.081 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308805813 0 0) 0x7feaf01d7c20 con 0x7feaf009ac40 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): my global_id is 4161 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf01d7db0 con 0x7feaf009ac40 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002040854 0 0) 0x7feaf01d7f40 con 0x7feaf01d77e0 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf01d7c20 con 0x7feaf01d77e0 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2060213315 0 0) 0x7fead4004eb0 con 0x7feaf00ad140 2024-03-29T10:39:16.082 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feaf01d7f40 con 0x7feaf00ad140 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7feae00017e0 con 0x7feaf00ad140 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:16.083 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3071836759 0 0) 0x7feaf01d7db0 con 0x7feaf009ac40 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fead4004eb0 con 0x7feaf009ac40 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7feaec001b00 con 0x7feaf009ac40 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:16.084 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3321736349 0 0) 0x7feaf01d7f40 con 0x7feaf00ad140 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 >> v1:172.21.15.114:6790/0 conn(0x7feaf01d77e0 legacy=0x7feaf01e1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 >> v1:172.21.15.114:6789/0 conn(0x7feaf009ac40 legacy=0x7feaf00b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 monclient: found mon.b 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf01e6390 con 0x7feaf00ad140 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:16.085 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feaf7360640 5 monclient: authenticate success, global_id 4159 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feaf7360640 10 monclient: _renew_subs 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feaf7360640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feaf7360640 1 -- 172.21.15.176:0/875045235 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf01e3360 con 0x7feaf00ad140 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feae0001d50 con 0x7feaf00ad140 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7feae00037b0 con 0x7feaf00ad140 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 monclient: dump: 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.086 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae77fe640 4 set_mon_vals no callback set 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feaf7360640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feaf7360640 10 monclient: _renew_subs 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feaf7360640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feaf7360640 1 -- 172.21.15.176:0/875045235 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf01e38e0 con 0x7feaf00ad140 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7feae0003ca0 con 0x7feaf00ad140 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feaf7360640 1 librados: init done 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 78639199 0 0) 0x7feae00057c0 con 0x7feaf00ad140 2024-03-29T10:39:16.087 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 10 client.4159.objecter ms_dispatch 0x7feaf0000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:39:16.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:39:16.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.082+0000 7feae5ffb640 3 client.4159.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:39:16.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.083+0000 7feae5ffb640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-29T10:39:16.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.083+0000 7feaf7360640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:16.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.083+0000 7feaf7360640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:16.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.083+0000 7feaf7360640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:16.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.083+0000 7feaf7360640 1 -- 172.21.15.176:0/875045235 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf01d7120 con 0x7feaf00ad140 2024-03-29T10:39:16.088 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:16.084+0000 7fdf5662b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:39:16.088 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.085+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7feae0005980 con 0x7feaf00ad140 2024-03-29T10:39:16.089 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.085+0000 7feae5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:16.089 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.085+0000 7feae5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:16.142 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.158+0000 7feaf7360640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1164e467-12f7-40e9-9770-7e27c30cfa84", "id": 7}] 2024-03-29T10:39:16.143 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.158+0000 7feaf7360640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1164e467-12f7-40e9-9770-7e27c30cfa84", "id": 7}] 2024-03-29T10:39:16.143 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.158+0000 7feaf7360640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:16.143 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.158+0000 7feaf7360640 1 -- 172.21.15.176:0/875045235 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1164e467-12f7-40e9-9770-7e27c30cfa84", "id": 7} v 0) v1 -- 0x7feaf009b020 con 0x7feaf00ad140 2024-03-29T10:39:16.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.161+0000 7feae5ffb640 1 -- 172.21.15.176:0/875045235 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1164e467-12f7-40e9-9770-7e27c30cfa84", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2013509708 0 1371665406) 0x7feae0003f50 con 0x7feaf00ad140 2024-03-29T10:39:16.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.161+0000 7feae5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1164e467-12f7-40e9-9770-7e27c30cfa84", "id": 7}] 2024-03-29T10:39:16.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.161+0000 7feae5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:16.145 INFO:teuthology.orchestra.run.smithi176.stdout:7 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 10 librados: watch_flush enter 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 10 librados: watch_flush exit 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 10 client.4159.objecter successfully canceled tick 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 10 monclient: shutdown 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 1 -- 172.21.15.176:0/875045235 >> v1:172.21.15.176:6789/0 conn(0x7feaf00ad140 legacy=0x7feaf00afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 1 -- 172.21.15.176:0/875045235 shutdown_connections 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 1 -- 172.21.15.176:0/875045235 >> 172.21.15.176:0/875045235 conn(0x7feaf0066b00 msgr2=0x7feaf013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 1 -- 172.21.15.176:0/875045235 shutdown_connections 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 1 -- 172.21.15.176:0/875045235 wait complete. 2024-03-29T10:39:16.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T10:39:16.162+0000 7feaf7360640 1 librados: shutdown 2024-03-29T10:39:16.153 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:16.169+0000 7fbb4763e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:39:16.156 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:39:16.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:39:16.159 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:39:16.159 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:39:16.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:39:16.161 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:39:16.161 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:39:16.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:39:16.163 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:39:16.163 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:39:16.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:39:16.165 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:39:16.165 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:39:16.165 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:39:16.167 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:39:16.168 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:39:16.168 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:39:16.169 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:39:16.170 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:39:16.170 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:39:16.172 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:39:16.172 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:39:16.172 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:39:16.174 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:39:16.175 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:16.225 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:16.241+0000 7fdf5662b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:39:16.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.307+0000 7f7fe316b640 10 monclient: get_monmap_and_config 2024-03-29T10:39:16.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.307+0000 7f7fe316b640 10 monclient: build_initial_monmap 2024-03-29T10:39:16.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.307+0000 7f7fe316b640 10 monclient: monmap: 2024-03-29T10:39:16.291 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-03-29T10:39:16.291 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.291 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:39:16.308426+0000 2024-03-29T10:39:16.292 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:39:16.308426+0000 2024-03-29T10:39:16.292 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:16.292 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:16.292 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:16.292 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:16.292 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:16.292 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:16.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.307+0000 7f7fe316b640 1 Processor -- start 2024-03-29T10:39:16.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.307+0000 7f7fe316b640 1 -- start start 2024-03-29T10:39:16.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.307+0000 7f7fe316b640 10 monclient: init 2024-03-29T10:39:16.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe316b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:16.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe316b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:16.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe316b640 10 monclient(hunting): picked mon.noname-b con 0x7f7fdc09ac40 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe316b640 10 monclient(hunting): picked mon.noname-c con 0x7f7fdc09c870 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe316b640 10 monclient(hunting): picked mon.noname-a con 0x7f7fdc0b4360 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe316b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc09dec0 con 0x7f7fdc0b4360 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe1708640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7fdc0b4360 0x7f7fdc14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37984/0 (socket says 172.21.15.114:37984) 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe1708640 1 -- 172.21.15.114:0/3945709864 learned_addr learned my addr 172.21.15.114:0/3945709864 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe0f07640 1 --1- 172.21.15.114:0/3945709864 >> v1:172.21.15.114:6790/0 conn(0x7f7fdc09ac40 0x7f7fdc0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42290/0 (socket says 172.21.15.114:42290) 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe316b640 1 -- 172.21.15.114:0/3945709864 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc09da30 con 0x7f7fdc09c870 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe316b640 1 -- 172.21.15.114:0/3945709864 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc0b39c0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe316b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fe316b640 10 monclient(hunting): authenticate will time out at 821.549787s 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058122187 0 0) 0x7f7fd4001490 con 0x7f7fdc0b4360 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fcc003520 con 0x7f7fdc0b4360 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153921270 0 0) 0x7f7fdc0b39c0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 10 monclient(hunting): my global_id is 4178 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.308+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd4001490 con 0x7f7fdc09ac40 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273307096 0 0) 0x7f7fdc09da30 con 0x7f7fdc09c870 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fdc0b39c0 con 0x7f7fdc09c870 2024-03-29T10:39:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1284952878 0 0) 0x7f7fcc003520 con 0x7f7fdc0b4360 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 >> v1:172.21.15.114:6790/0 conn(0x7f7fdc09ac40 legacy=0x7f7fdc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 >> v1:172.21.15.176:6789/0 conn(0x7f7fdc09c870 legacy=0x7f7fdc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 1 monclient: found mon.noname-a 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc0b5160 con 0x7f7fdc0b4360 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7fd4001a00 con 0x7f7fdc0b4360 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient: dump: 2024-03-29T10:39:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fe316b640 5 monclient: authenticate success, global_id 4188 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fe316b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fd4001b60 con 0x7f7fdc0b4360 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 1 -- 172.21.15.114:0/3945709864 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7fd4001fb0 con 0x7f7fdc0b4360 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fe316b640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fe316b640 4 set_mon_vals no callback set 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fdb7fe640 10 monclient: dump: 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:16.307 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fe316b640 10 monclient: shutdown 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fe316b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fe316b640 1 -- 172.21.15.114:0/3945709864 >> v1:172.21.15.114:6789/0 conn(0x7f7fdc0b4360 legacy=0x7f7fdc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fe316b640 1 -- 172.21.15.114:0/3945709864 shutdown_connections 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fe316b640 1 -- 172.21.15.114:0/3945709864 >> 172.21.15.114:0/3945709864 conn(0x7f7fdc066b00 msgr2=0x7f7fdc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fe316b640 1 -- 172.21.15.114:0/3945709864 shutdown_connections 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.309+0000 7f7fe316b640 1 -- 172.21.15.114:0/3945709864 wait complete. 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.310+0000 7f7fe316b640 10 monclient: build_initial_monmap 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.310+0000 7f7fe316b640 10 monclient: monmap: 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-03-29T10:39:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.310+0000 7f7fe316b640 1 librados: starting msgr at 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.310+0000 7f7fe316b640 1 librados: starting objecter 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe316b640 1 Processor -- start 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe316b640 1 -- start start 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe316b640 1 librados: setting wanted keys 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe316b640 1 librados: calling monclient init 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe316b640 10 monclient: init 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe316b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe316b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:16.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe316b640 10 monclient(hunting): picked mon.noname-a con 0x7f7fdc09ac40 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe316b640 10 monclient(hunting): picked mon.noname-c con 0x7f7fdc1d7250 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe316b640 10 monclient(hunting): picked mon.noname-b con 0x7f7fdc1d7700 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe0f07640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7fdc09ac40 0x7f7fdc1411b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37996/0 (socket says 172.21.15.114:37996) 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.311+0000 7f7fe0f07640 1 -- 172.21.15.114:0/1641068767 learned_addr learned my addr 172.21.15.114:0/1641068767 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc1d7b70 con 0x7f7fdc09ac40 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc1397e0 con 0x7f7fdc1d7700 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 client.?.objecter ms_handle_connect 0x7f7fdc09ac40 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc142a60 con 0x7f7fdc1d7250 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 client.?.objecter ms_handle_connect 0x7f7fdc1d7250 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 10 monclient(hunting): authenticate will time out at 821.553190s 2024-03-29T10:39:16.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673388394 0 0) 0x7f7fdc1d7b70 con 0x7f7fdc09ac40 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc0004eb0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 client.?.objecter ms_handle_connect 0x7f7fdc1d7700 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855820024 0 0) 0x7f7fdc142a60 con 0x7f7fdc1d7250 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): my global_id is 4181 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fdc1d7b70 con 0x7f7fdc1d7250 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086243693 0 0) 0x7f7fdc1397e0 con 0x7f7fdc1d7700 2024-03-29T10:39:16.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): my global_id is 4186 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fdc142a60 con 0x7f7fdc1d7700 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2315659164 0 0) 0x7f7fc0004eb0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fdc1397e0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7fd0001270 con 0x7f7fdc09ac40 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): dump: 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.312 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2372874386 0 0) 0x7f7fdc1d7b70 con 0x7f7fdc1d7250 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc0004eb0 con 0x7f7fdc1d7250 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2022094568 0 0) 0x7f7fdc142a60 con 0x7f7fdc1d7700 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fdc1d7b70 con 0x7f7fdc1d7700 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7fc8001870 con 0x7f7fdc1d7250 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): dump: 2024-03-29T10:39:16.313 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7fd4001e70 con 0x7f7fdc1d7700 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient(hunting): dump: 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T10:39:16.314 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2576525328 0 0) 0x7f7fdc1397e0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 >> v1:172.21.15.114:6790/0 conn(0x7f7fdc1d7250 legacy=0x7f7fdc1389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 >> v1:172.21.15.176:6789/0 conn(0x7f7fdc1d7700 legacy=0x7f7fdc1390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 monclient: found mon.a 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc1d6d30 con 0x7f7fdc09ac40 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient: _finish_auth 0 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 5 monclient: authenticate success, global_id 4191 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 10 monclient: _renew_subs 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fdc1418c0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fd00047e0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 10 monclient: _renew_subs 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fdc0af970 con 0x7f7fdc09ac40 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.312+0000 7f7fd97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7fd0004af0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fd97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fd97fa640 10 monclient: dump: 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fdaffd640 4 set_mon_vals no callback set 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fe316b640 1 librados: init done 2024-03-29T10:39:16.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7fd0004fe0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f7fd00069b0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fd97fa640 10 client.4191.objecter ms_dispatch 0x7f7fdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:16.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fd97fa640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:16.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fd97fa640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:16.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fd97fa640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T10:39:16.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fe316b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:16.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fe316b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:16.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fe316b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:16.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.313+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc074bb0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.319 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:16.315+0000 7fbb4763e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:39:16.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.316+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7fdc074bb0 con 0x7f7fdc09ac40 2024-03-29T10:39:16.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.316+0000 7f7fd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:16.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.316+0000 7f7fd97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:16.347 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:16.362+0000 7fbb4763e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:39:16.359 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:16.375+0000 7fdf5662b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:39:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.377+0000 7f7fe316b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.377+0000 7f7fe316b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.377+0000 7f7fe316b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.377+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fdc09b020 con 0x7f7fdc09ac40 2024-03-29T10:39:16.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.378+0000 7f7fd97fa640 1 -- 172.21.15.114:0/1641068767 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f7fd0005c20 con 0x7f7fdc09ac40 2024-03-29T10:39:16.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.378+0000 7f7fd97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:16.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.378+0000 7f7fd97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:16.362 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T10:39:16.362 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:16.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.378+0000 7f7fe316b640 10 librados: watch_flush enter 2024-03-29T10:39:16.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.378+0000 7f7fe316b640 10 librados: watch_flush exit 2024-03-29T10:39:16.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.378+0000 7f7fe316b640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T10:39:16.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.378+0000 7f7fe316b640 10 client.4191.objecter successfully canceled tick 2024-03-29T10:39:16.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.378+0000 7f7fe316b640 10 monclient: shutdown 2024-03-29T10:39:16.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.378+0000 7f7fe316b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:16.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.378+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 >> v1:172.21.15.114:6789/0 conn(0x7f7fdc09ac40 legacy=0x7f7fdc1411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:16.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.379+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 shutdown_connections 2024-03-29T10:39:16.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.379+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 >> 172.21.15.114:0/1641068767 conn(0x7f7fdc066b00 msgr2=0x7f7fdc13a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:16.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.379+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 shutdown_connections 2024-03-29T10:39:16.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.379+0000 7f7fe316b640 1 -- 172.21.15.114:0/1641068767 wait complete. 2024-03-29T10:39:16.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T10:39:16.379+0000 7f7fe316b640 1 librados: shutdown 2024-03-29T10:39:16.373 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:39:16.373 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:39:16.404 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:16.420+0000 7fdf5662b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:39:16.462 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:16.478+0000 7fbb4763e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:39:16.517 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:16.533+0000 7fdf5662b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:39:16.557 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:16.573+0000 7fbb4763e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:39:16.613 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:16.629+0000 7fdf5662b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:39:16.711 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:16.727+0000 7fbb4763e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:39:16.753 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:16.768+0000 7fbb4763e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:39:16.765 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:16.781+0000 7fdf5662b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:39:16.775 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-03-29T10:39:16.792+0000 7fb5e9611640 -1 Falling back to public interface 2024-03-29T10:39:16.776 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-03-29T10:39:16.792+0000 7fcdd6013640 -1 Falling back to public interface 2024-03-29T10:39:16.783 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-03-29T10:39:16.799+0000 7f639f611640 -1 Falling back to public interface 2024-03-29T10:39:16.783 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-03-29T10:39:16.799+0000 7f8497c89640 -1 Falling back to public interface 2024-03-29T10:39:16.794 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-03-29T10:39:16.810+0000 7f0d4260c640 -1 Falling back to public interface 2024-03-29T10:39:16.795 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-03-29T10:39:16.810+0000 7f47a4610640 -1 Falling back to public interface 2024-03-29T10:39:16.795 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-03-29T10:39:16.810+0000 7f0d40fd9640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6804/0: (98) Address already in use 2024-03-29T10:39:16.795 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-03-29T10:39:16.811+0000 7fe8e5462640 -1 Falling back to public interface 2024-03-29T10:39:16.797 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-03-29T10:39:16.813+0000 7fcaa8bfa640 -1 Falling back to public interface 2024-03-29T10:39:16.812 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:16.828+0000 7fdf5662b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:39:16.913 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:16.929+0000 7fbb4763e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:39:16.964 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:16.980+0000 7fdf5662b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:39:17.178 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:17.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.281+0000 7f0dba7c1640 10 monclient: get_monmap_and_config 2024-03-29T10:39:17.266 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.281+0000 7f0dba7c1640 10 monclient: build_initial_monmap 2024-03-29T10:39:17.266 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.281+0000 7f0dba7c1640 10 monclient: monmap: 2024-03-29T10:39:17.266 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:17.266 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:17.266 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:39:17.282937+0000 2024-03-29T10:39:17.266 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:39:17.282937+0000 2024-03-29T10:39:17.266 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:17.266 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:17.266 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:17.266 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:17.266 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:17.267 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:17.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.282+0000 7f0dba7c1640 1 Processor -- start 2024-03-29T10:39:17.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.282+0000 7f0dba7c1640 1 -- start start 2024-03-29T10:39:17.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.282+0000 7f0dba7c1640 10 monclient: init 2024-03-29T10:39:17.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:17.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:17.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 10 monclient(hunting): picked mon.noname-a con 0x7f0db41457f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:17.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 10 monclient(hunting): picked mon.noname-b con 0x7f0db413f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:17.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 10 monclient(hunting): picked mon.noname-c con 0x7f0db40a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db40aaef0 con 0x7f0db41457f0 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db4145bd0 con 0x7f0db40a9d70 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db4145eb0 con 0x7f0db413f650 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 10 monclient(hunting): authenticate will time out at 822.524123s 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db37fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f0db413f650 0x7f0db40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42298/0 (socket says 172.21.15.114:42298) 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db37fe640 1 -- 172.21.15.114:0/1863388010 learned_addr learned my addr 172.21.15.114:0/1863388010 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377181371 0 0) 0x7f0db4145eb0 con 0x7f0db413f650 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): my global_id is 4184 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c003520 con 0x7f0db413f650 2024-03-29T10:39:17.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043680491 0 0) 0x7f0db40aaef0 con 0x7f0db41457f0 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db4145eb0 con 0x7f0db41457f0 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732679644 0 0) 0x7f0db4145bd0 con 0x7f0db40a9d70 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db40aaef0 con 0x7f0db40a9d70 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1911109563 0 0) 0x7f0d9c003520 con 0x7f0db413f650 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 >> v1:172.21.15.176:6789/0 conn(0x7f0db40a9d70 legacy=0x7f0db414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:17.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 >> v1:172.21.15.114:6789/0 conn(0x7f0db41457f0 legacy=0x7f0db413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 monclient: found mon.noname-b 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db4146340 con 0x7f0db413f650 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 5 monclient: authenticate success, global_id 4184 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0da4001690 con 0x7f0db413f650 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:17.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient: dump: 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0da4001870 con 0x7f0db413f650 2024-03-29T10:39:17.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 1 -- 172.21.15.114:0/1863388010 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0da4001cc0 con 0x7f0db413f650 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0dba7c1640 4 set_mon_vals no callback set 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.283+0000 7f0db2ffd640 10 monclient: dump: 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:17.280 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.284+0000 7f0dba7c1640 10 monclient: shutdown 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.284+0000 7f0dba7c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.284+0000 7f0dba7c1640 1 -- 172.21.15.114:0/1863388010 >> v1:172.21.15.114:6790/0 conn(0x7f0db413f650 legacy=0x7f0db40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.284+0000 7f0dba7c1640 1 -- 172.21.15.114:0/1863388010 shutdown_connections 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.284+0000 7f0dba7c1640 1 -- 172.21.15.114:0/1863388010 >> 172.21.15.114:0/1863388010 conn(0x7f0db4066a00 msgr2=0x7f0db4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.284+0000 7f0dba7c1640 1 -- 172.21.15.114:0/1863388010 shutdown_connections 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.284+0000 7f0dba7c1640 1 -- 172.21.15.114:0/1863388010 wait complete. 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.285+0000 7f0dba7c1640 10 monclient: build_initial_monmap 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.285+0000 7f0dba7c1640 10 monclient: monmap: 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:39:17.281 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.285+0000 7f0dba7c1640 1 librados: starting msgr at 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.285+0000 7f0dba7c1640 1 librados: starting objecter 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.285+0000 7f0dba7c1640 1 Processor -- start 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.285+0000 7f0dba7c1640 1 -- start start 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.285+0000 7f0dba7c1640 1 librados: setting wanted keys 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.285+0000 7f0dba7c1640 1 librados: calling monclient init 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.285+0000 7f0dba7c1640 10 monclient: init 2024-03-29T10:39:17.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0dba7c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0dba7c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0dba7c1640 10 monclient(hunting): picked mon.noname-c con 0x7f0db40a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0dba7c1640 10 monclient(hunting): picked mon.noname-a con 0x7f0db413f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0dba7c1640 10 monclient(hunting): picked mon.noname-b con 0x7f0db41457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0dba7c1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db40ac1e0 con 0x7f0db413f650 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0dba7c1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db41ce3a0 con 0x7f0db41457f0 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0dba7c1640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db41ce530 con 0x7f0db40a9d70 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0dba7c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0dba7c1640 10 monclient(hunting): authenticate will time out at 822.527281s 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db37fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0db413f650 0x7f0db41cb340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38014/0 (socket says 172.21.15.114:38014) 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db37fe640 1 -- 172.21.15.114:0/866715452 learned_addr learned my addr 172.21.15.114:0/866715452 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db3fff640 1 --1- 172.21.15.114:0/866715452 >> v1:172.21.15.114:6790/0 conn(0x7f0db40a9d70 0x7f0db41cdb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42310/0 (socket says 172.21.15.114:42310) 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 client.?.objecter ms_handle_connect 0x7f0db413f650 2024-03-29T10:39:17.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 client.?.objecter ms_handle_connect 0x7f0db40a9d70 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 client.?.objecter ms_handle_connect 0x7f0db41457f0 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963658633 0 0) 0x7f0db40ac1e0 con 0x7f0db413f650 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): my global_id is 4197 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da0004eb0 con 0x7f0db413f650 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826376835 0 0) 0x7f0db41ce530 con 0x7f0db40a9d70 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:17.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): my global_id is 4187 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db40ac1e0 con 0x7f0db40a9d70 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728589177 0 0) 0x7f0db41ce3a0 con 0x7f0db41457f0 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): my global_id is 4192 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db41ce530 con 0x7f0db41457f0 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4192618187 0 0) 0x7f0da0004eb0 con 0x7f0db413f650 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0db41ce3a0 con 0x7f0db413f650 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0da4001bc0 con 0x7f0db413f650 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:17.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1112259491 0 0) 0x7f0db40ac1e0 con 0x7f0db40a9d70 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0da0004eb0 con 0x7f0db40a9d70 2024-03-29T10:39:17.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0da80017f0 con 0x7f0db40a9d70 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.286+0000 7f0db0ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1774860786 0 0) 0x7f0db41ce530 con 0x7f0db41457f0 2024-03-29T10:39:17.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0db40ac1e0 con 0x7f0db41457f0 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0dac001c70 con 0x7f0db41457f0 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:17.288 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2278765034 0 0) 0x7f0db41ce3a0 con 0x7f0db413f650 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 >> v1:172.21.15.114:6790/0 conn(0x7f0db40a9d70 legacy=0x7f0db41cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 >> v1:172.21.15.176:6789/0 conn(0x7f0db41457f0 legacy=0x7f0db41e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 monclient: found mon.a 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db41cba50 con 0x7f0db413f650 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0dba7c1640 5 monclient: authenticate success, global_id 4197 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0dba7c1640 10 monclient: _renew_subs 2024-03-29T10:39:17.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0dba7c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0dba7c1640 1 -- 172.21.15.114:0/866715452 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db41cbcc0 con 0x7f0db413f650 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0dba7c1640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0dba7c1640 10 monclient: _renew_subs 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0dba7c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0dba7c1640 1 -- 172.21.15.114:0/866715452 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db41cef30 con 0x7f0db413f650 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0dba7c1640 1 librados: init done 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0da4004bf0 con 0x7f0db413f650 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0da4004f00 con 0x7f0db413f650 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 monclient: dump: 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:17.290 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db27fc640 4 set_mon_vals no callback set 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0da40053f0 con 0x7f0db413f650 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f0da4006ac0 con 0x7f0db413f650 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 10 client.4197.objecter ms_dispatch 0x7f0db4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 3 client.4197.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 3 client.4197.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:17.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.287+0000 7f0db0ff9640 20 client.4197.objecter dump_active .. 0 homeless 2024-03-29T10:39:17.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.288+0000 7f0dba7c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:17.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.288+0000 7f0dba7c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:17.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.288+0000 7f0dba7c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:17.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.288+0000 7f0dba7c1640 1 -- 172.21.15.114:0/866715452 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db4075f10 con 0x7f0db413f650 2024-03-29T10:39:17.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.290+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0da40064a0 con 0x7f0db413f650 2024-03-29T10:39:17.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.290+0000 7f0db0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:17.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.290+0000 7f0db0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:17.336 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.352+0000 7f0dba7c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:17.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.352+0000 7f0dba7c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:17.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.352+0000 7f0dba7c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:17.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.352+0000 7f0dba7c1640 1 -- 172.21.15.114:0/866715452 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0db40b3680 con 0x7f0db413f650 2024-03-29T10:39:17.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.352+0000 7f0db0ff9640 1 -- 172.21.15.114:0/866715452 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f0da40066f0 con 0x7f0db413f650 2024-03-29T10:39:17.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.352+0000 7f0db0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:17.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.352+0000 7f0db0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:17.337 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:39:17.337 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:17.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.353+0000 7f0dba7c1640 10 librados: watch_flush enter 2024-03-29T10:39:17.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.353+0000 7f0dba7c1640 10 librados: watch_flush exit 2024-03-29T10:39:17.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.353+0000 7f0dba7c1640 20 client.4197.objecter shutdown clearing up homeless session... 2024-03-29T10:39:17.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.353+0000 7f0dba7c1640 10 client.4197.objecter successfully canceled tick 2024-03-29T10:39:17.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.353+0000 7f0dba7c1640 10 monclient: shutdown 2024-03-29T10:39:17.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.353+0000 7f0dba7c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:17.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.353+0000 7f0dba7c1640 1 -- 172.21.15.114:0/866715452 >> v1:172.21.15.114:6789/0 conn(0x7f0db413f650 legacy=0x7f0db41cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:17.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.353+0000 7f0dba7c1640 1 -- 172.21.15.114:0/866715452 shutdown_connections 2024-03-29T10:39:17.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.353+0000 7f0dba7c1640 1 -- 172.21.15.114:0/866715452 >> 172.21.15.114:0/866715452 conn(0x7f0db4066a00 msgr2=0x7f0db40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:17.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.354+0000 7f0dba7c1640 1 -- 172.21.15.114:0/866715452 shutdown_connections 2024-03-29T10:39:17.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.354+0000 7f0dba7c1640 1 -- 172.21.15.114:0/866715452 wait complete. 2024-03-29T10:39:17.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:17.354+0000 7f0dba7c1640 1 librados: shutdown 2024-03-29T10:39:17.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:17.733 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:17.749+0000 7fbb4763e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:39:17.789 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:17.805+0000 7fdf5662b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:39:18.008 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:18.024+0000 7fbb4763e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:39:18.059 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:18.075+0000 7fbb4763e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:39:18.063 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:18.079+0000 7fdf5662b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:39:18.114 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:18.130+0000 7fdf5662b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:39:18.174 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:18.190+0000 7fbb4763e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:39:18.236 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T10:39:18.252+0000 7fdf5662b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:39:19.188 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:19.204+0000 7fbae823d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:39:19.189 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T10:39:19.204+0000 7fbae823d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:39:20.236 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-03-29T10:39:20.252+0000 7f8497c89640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:39:20.259 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-03-29T10:39:20.275+0000 7f639f611640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:39:20.298 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-03-29T10:39:20.314+0000 7fb5e9611640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:39:20.301 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-03-29T10:39:20.317+0000 7fcaa8bfa640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:39:20.306 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-03-29T10:39:20.322+0000 7fcdd6013640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:39:20.341 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-03-29T10:39:20.357+0000 7f0d4260c640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:39:20.344 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-03-29T10:39:20.360+0000 7fe8e5462640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:39:20.351 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-03-29T10:39:20.367+0000 7f47a4610640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:39:24.154 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:24.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.318+0000 7f2721120640 10 monclient: get_monmap_and_config 2024-03-29T10:39:24.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.318+0000 7f2721120640 10 monclient: build_initial_monmap 2024-03-29T10:39:24.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.318+0000 7f2721120640 10 monclient: monmap: 2024-03-29T10:39:24.303 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:24.303 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:24.303 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:39:24.319768+0000 2024-03-29T10:39:24.303 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:39:24.319768+0000 2024-03-29T10:39:24.304 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:24.304 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:24.304 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:24.304 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:24.304 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:24.304 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:24.304 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.319+0000 7f2721120640 1 Processor -- start 2024-03-29T10:39:24.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.319+0000 7f2721120640 1 -- start start 2024-03-29T10:39:24.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.319+0000 7f2721120640 10 monclient: init 2024-03-29T10:39:24.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f2721120640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:24.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f2721120640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:24.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f2721120640 10 monclient(hunting): picked mon.noname-c con 0x7f271c0b0770 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:24.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f2721120640 10 monclient(hunting): picked mon.noname-b con 0x7f271c14a700 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:24.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f2721120640 10 monclient(hunting): picked mon.noname-a con 0x7f271c0ac5a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:24.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f2721120640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271c0aba70 con 0x7f271c0ac5a0 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f2721120640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271c0aecb0 con 0x7f271c0b0770 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f2721120640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271c0af150 con 0x7f271c14a700 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f2721120640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f2721120640 10 monclient(hunting): authenticate will time out at 829.561427s 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f271a59c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f271c14a700 0x7f271c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:50666/0 (socket says 172.21.15.114:50666) 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.320+0000 7f271a59c640 1 -- 172.21.15.114:0/3618369931 learned_addr learned my addr 172.21.15.114:0/3618369931 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545762351 0 0) 0x7f271c0af150 con 0x7f271c14a700 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 10 monclient(hunting): my global_id is 4226 2024-03-29T10:39:24.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2704003520 con 0x7f271c14a700 2024-03-29T10:39:24.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245826629 0 0) 0x7f271c0aba70 con 0x7f271c0ac5a0 2024-03-29T10:39:24.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:39:24.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:24.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:24.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:39:24.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.321+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f271c0af150 con 0x7f271c0ac5a0 2024-03-29T10:39:24.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213644069 0 0) 0x7f271c0aecb0 con 0x7f271c0b0770 2024-03-29T10:39:24.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:39:24.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:24.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:24.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f271c0aba70 con 0x7f271c0b0770 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3858449665 0 0) 0x7f2704003520 con 0x7f271c14a700 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 >> v1:172.21.15.176:6789/0 conn(0x7f271c0b0770 legacy=0x7f271c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 >> v1:172.21.15.114:6789/0 conn(0x7f271c0ac5a0 legacy=0x7f271c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 1 monclient: found mon.noname-b 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c0ac9d0 con 0x7f271c14a700 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient: _finish_auth 0 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:39:24.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2721120640 5 monclient: authenticate success, global_id 4226 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2721120640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2710001690 con 0x7f271c14a700 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient: dump: 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:24.310 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2721120640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2710001870 con 0x7f271c14a700 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 1 -- 172.21.15.114:0/3618369931 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2710001690 con 0x7f271c14a700 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2719d9b640 10 monclient: dump: 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:24.311 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2721120640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.322+0000 7f2721120640 4 set_mon_vals no callback set 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.323+0000 7f2721120640 10 monclient: shutdown 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.323+0000 7f2721120640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.323+0000 7f2721120640 1 -- 172.21.15.114:0/3618369931 >> v1:172.21.15.114:6790/0 conn(0x7f271c14a700 legacy=0x7f271c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.323+0000 7f2721120640 1 -- 172.21.15.114:0/3618369931 shutdown_connections 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.323+0000 7f2721120640 1 -- 172.21.15.114:0/3618369931 >> 172.21.15.114:0/3618369931 conn(0x7f271c06b660 msgr2=0x7f271c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.323+0000 7f2721120640 1 -- 172.21.15.114:0/3618369931 shutdown_connections 2024-03-29T10:39:24.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.323+0000 7f2721120640 1 -- 172.21.15.114:0/3618369931 wait complete. 2024-03-29T10:39:24.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.324+0000 7f2721120640 10 monclient: build_initial_monmap 2024-03-29T10:39:24.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.324+0000 7f2721120640 10 monclient: monmap: 2024-03-29T10:39:24.316 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:24.316 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:24.316 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:39:24.316 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:39:24.316 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:24.316 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:24.316 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:24.316 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.325+0000 7f2721120640 1 librados: starting msgr at 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.325+0000 7f2721120640 1 librados: starting objecter 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.325+0000 7f2721120640 1 Processor -- start 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.325+0000 7f2721120640 1 -- start start 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.325+0000 7f2721120640 1 librados: setting wanted keys 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.325+0000 7f2721120640 1 librados: calling monclient init 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.325+0000 7f2721120640 10 monclient: init 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.326+0000 7f2721120640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.326+0000 7f2721120640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.326+0000 7f2721120640 10 monclient(hunting): picked mon.noname-a con 0x7f271c0ac5a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.326+0000 7f271ad9d640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f271c0ac5a0 0x7f271c1e7b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48040/0 (socket says 172.21.15.114:48040) 2024-03-29T10:39:24.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.326+0000 7f271ad9d640 1 -- 172.21.15.114:0/2576970316 learned_addr learned my addr 172.21.15.114:0/2576970316 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.326+0000 7f2721120640 10 monclient(hunting): picked mon.noname-b con 0x7f271c0b0770 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.326+0000 7f2721120640 10 monclient(hunting): picked mon.noname-c con 0x7f271c11b1c0 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.326+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271c1e76d0 con 0x7f271c0ac5a0 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271c11dae0 con 0x7f271c0b0770 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271c1e8270 con 0x7f271c11b1c0 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f2721120640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f2721120640 10 monclient(hunting): authenticate will time out at 829.568136s 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 client.?.objecter ms_handle_connect 0x7f271c0ac5a0 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 client.?.objecter ms_handle_connect 0x7f271c0b0770 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544937726 0 0) 0x7f271c11dae0 con 0x7f271c0b0770 2024-03-29T10:39:24.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:24.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:24.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:24.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:39:24.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2708004eb0 con 0x7f271c0b0770 2024-03-29T10:39:24.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418540385 0 0) 0x7f271c1e76d0 con 0x7f271c0ac5a0 2024-03-29T10:39:24.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:24.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:24.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:24.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f271c11dae0 con 0x7f271c0ac5a0 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 client.?.objecter ms_handle_connect 0x7f271c11b1c0 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.327+0000 7f27037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415240964 0 0) 0x7f271c1e8270 con 0x7f271c11b1c0 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 10 monclient(hunting): my global_id is 4229 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f271c1e76d0 con 0x7f271c11b1c0 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1475985107 0 0) 0x7f2708004eb0 con 0x7f271c0b0770 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f271c1e8270 con 0x7f271c0b0770 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2710001690 con 0x7f271c0b0770 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:24.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 10 monclient(hunting): dump: 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.328+0000 7f27037fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 191054908 0 0) 0x7f271c11dae0 con 0x7f271c0ac5a0 2024-03-29T10:39:24.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2708004eb0 con 0x7f271c0ac5a0 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f270c001990 con 0x7f271c0ac5a0 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 10 monclient(hunting): dump: 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:24.322 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1509745514 0 0) 0x7f271c1e8270 con 0x7f271c0b0770 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 >> v1:172.21.15.114:6790/0 conn(0x7f271c11b1c0 legacy=0x7f271c1eea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 >> v1:172.21.15.114:6789/0 conn(0x7f271c0ac5a0 legacy=0x7f271c1e7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 1 monclient: found mon.b 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c1e8400 con 0x7f271c0b0770 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f2721120640 5 monclient: authenticate success, global_id 4237 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f2721120640 10 monclient: _renew_subs 2024-03-29T10:39:24.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f2721120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:24.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271c11ecb0 con 0x7f271c0b0770 2024-03-29T10:39:24.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f2721120640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:24.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f2721120640 10 monclient: _renew_subs 2024-03-29T10:39:24.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f2721120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:24.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f271c1f01b0 con 0x7f271c0b0770 2024-03-29T10:39:24.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f2721120640 1 librados: init done 2024-03-29T10:39:24.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2710001000 con 0x7f271c0b0770 2024-03-29T10:39:24.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f27037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:24.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.329+0000 7f271959a640 4 set_mon_vals no callback set 2024-03-29T10:39:24.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.330+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2710004b40 con 0x7f271c0b0770 2024-03-29T10:39:24.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.330+0000 7f27037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.330+0000 7f27037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.330+0000 7f27037fe640 10 monclient: dump: 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:24.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.331+0000 7f2721120640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:24.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.332+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f271001ab10 con 0x7f271c0b0770 2024-03-29T10:39:24.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.331+0000 7f2721120640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:24.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.332+0000 7f2721120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:24.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.332+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc005080 con 0x7f271c0b0770 2024-03-29T10:39:24.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.332+0000 7f271ad9d640 10 monclient: get_auth_request con 0x7f270806df90 auth_method 0 2024-03-29T10:39:24.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.332+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f2710084ab0 con 0x7f271c0b0770 2024-03-29T10:39:24.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.332+0000 7f27037fe640 10 client.4237.objecter ms_dispatch 0x7f271c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:24.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.332+0000 7f27037fe640 3 client.4237.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:24.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.332+0000 7f27037fe640 3 client.4237.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:24.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.333+0000 7f27037fe640 20 client.4237.objecter dump_active .. 0 homeless 2024-03-29T10:39:24.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.333+0000 7f27037fe640 10 client.4237.objecter ms_handle_connect 0x7f270806df90 2024-03-29T10:39:24.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.339+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2710050390 con 0x7f271c0b0770 2024-03-29T10:39:24.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.339+0000 7f27037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:24.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.339+0000 7f27037fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:24.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.727+0000 7f2721120640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:24.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.727+0000 7f2721120640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:24.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.727+0000 7f2721120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:24.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.727+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26fc005330 con 0x7f271c0b0770 2024-03-29T10:39:24.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.728+0000 7f27037fe640 1 -- 172.21.15.114:0/2576970316 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f27100522c0 con 0x7f271c0b0770 2024-03-29T10:39:24.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.728+0000 7f27037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:24.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.728+0000 7f27037fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:24.713 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:39:24.713 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:24.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.731+0000 7f2721120640 10 librados: watch_flush enter 2024-03-29T10:39:24.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.731+0000 7f2721120640 10 librados: watch_flush exit 2024-03-29T10:39:24.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.731+0000 7f2721120640 20 client.4237.objecter shutdown clearing up homeless session... 2024-03-29T10:39:24.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.731+0000 7f2721120640 10 client.4237.objecter successfully canceled tick 2024-03-29T10:39:24.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.731+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 >> v1:172.21.15.114:6816/775044307 conn(0x7f270806df90 legacy=0x7f2708070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:24.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.731+0000 7f2721120640 10 monclient: shutdown 2024-03-29T10:39:24.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.731+0000 7f2721120640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:24.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.731+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 >> v1:172.21.15.176:6789/0 conn(0x7f271c0b0770 legacy=0x7f271c11e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:24.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.732+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 shutdown_connections 2024-03-29T10:39:24.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.732+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 >> 172.21.15.114:0/2576970316 conn(0x7f271c06b660 msgr2=0x7f271c149540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:24.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.732+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 shutdown_connections 2024-03-29T10:39:24.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.732+0000 7f2721120640 1 -- 172.21.15.114:0/2576970316 wait complete. 2024-03-29T10:39:24.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:24.732+0000 7f2721120640 1 librados: shutdown 2024-03-29T10:39:24.730 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:31.538 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:31.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.709+0000 7fb0d4243640 10 monclient: get_monmap_and_config 2024-03-29T10:39:31.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.709+0000 7fb0d4243640 10 monclient: build_initial_monmap 2024-03-29T10:39:31.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.709+0000 7fb0d4243640 10 monclient: monmap: 2024-03-29T10:39:31.694 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:31.694 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:31.694 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:39:31.710939+0000 2024-03-29T10:39:31.695 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:39:31.710939+0000 2024-03-29T10:39:31.695 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:31.695 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:31.695 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:31.695 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:31.695 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:31.695 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:31.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.710+0000 7fb0d4243640 1 Processor -- start 2024-03-29T10:39:31.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.711+0000 7fb0d4243640 1 -- start start 2024-03-29T10:39:31.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.711+0000 7fb0d4243640 10 monclient: init 2024-03-29T10:39:31.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.711+0000 7fb0d4243640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:31.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.711+0000 7fb0d4243640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:31.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.711+0000 7fb0d4243640 10 monclient(hunting): picked mon.noname-b con 0x7fb0cc0b0770 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:31.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.711+0000 7fb0d4243640 10 monclient(hunting): picked mon.noname-c con 0x7fb0cc14a700 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:31.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.711+0000 7fb0d4243640 10 monclient(hunting): picked mon.noname-a con 0x7fb0cc0ac5a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:31.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.711+0000 7fb0d4243640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc0aba70 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.711+0000 7fb0d4243640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc0aecb0 con 0x7fb0cc14a700 2024-03-29T10:39:31.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.711+0000 7fb0d4243640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc0af150 con 0x7fb0cc0b0770 2024-03-29T10:39:31.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.712+0000 7fb0d4243640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:31.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.712+0000 7fb0d4243640 10 monclient(hunting): authenticate will time out at 836.953011s 2024-03-29T10:39:31.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.712+0000 7fb0d27e0640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb0cc0ac5a0 0x7fb0cc151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36636/0 (socket says 172.21.15.114:36636) 2024-03-29T10:39:31.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.712+0000 7fb0d27e0640 1 -- 172.21.15.114:0/127271501 learned_addr learned my addr 172.21.15.114:0/127271501 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:31.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.712+0000 7fb0d0fdd640 1 -- 172.21.15.114:0/127271501 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697513109 0 0) 0x7fb0cc0aba70 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.712+0000 7fb0d0fdd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.712+0000 7fb0d0fdd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.712+0000 7fb0d0fdd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.712+0000 7fb0d0fdd640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.712+0000 7fb0d0fdd640 1 -- 172.21.15.114:0/127271501 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b8003520 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 1 -- 172.21.15.114:0/127271501 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1532213887 0 0) 0x7fb0b8003520 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 1 -- 172.21.15.114:0/127271501 >> v1:172.21.15.114:6790/0 conn(0x7fb0cc0b0770 legacy=0x7fb0cc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 1 -- 172.21.15.114:0/127271501 >> v1:172.21.15.176:6789/0 conn(0x7fb0cc14a700 legacy=0x7fb0cc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 1 monclient: found mon.noname-a 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 1 -- 172.21.15.114:0/127271501 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc0ac9d0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 10 monclient: _finish_auth 0 2024-03-29T10:39:31.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 1 -- 172.21.15.114:0/127271501 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb0c80015c0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d4243640 5 monclient: authenticate success, global_id 4245 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d4243640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 10 monclient: dump: 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.700 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d4243640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 1 -- 172.21.15.114:0/127271501 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0c8001720 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 1 -- 172.21.15.114:0/127271501 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb0c8001b70 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:31.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d0fdd640 10 monclient: dump: 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d4243640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:31.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.713+0000 7fb0d4243640 4 set_mon_vals no callback set 2024-03-29T10:39:31.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.714+0000 7fb0d4243640 10 monclient: shutdown 2024-03-29T10:39:31.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.714+0000 7fb0d4243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:31.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.714+0000 7fb0d4243640 1 -- 172.21.15.114:0/127271501 >> v1:172.21.15.114:6789/0 conn(0x7fb0cc0ac5a0 legacy=0x7fb0cc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:31.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.714+0000 7fb0d4243640 1 -- 172.21.15.114:0/127271501 shutdown_connections 2024-03-29T10:39:31.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.714+0000 7fb0d4243640 1 -- 172.21.15.114:0/127271501 >> 172.21.15.114:0/127271501 conn(0x7fb0cc06b660 msgr2=0x7fb0cc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:31.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.715+0000 7fb0d4243640 1 -- 172.21.15.114:0/127271501 shutdown_connections 2024-03-29T10:39:31.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.715+0000 7fb0d4243640 1 -- 172.21.15.114:0/127271501 wait complete. 2024-03-29T10:39:31.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.715+0000 7fb0d4243640 10 monclient: build_initial_monmap 2024-03-29T10:39:31.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.715+0000 7fb0d4243640 10 monclient: monmap: 2024-03-29T10:39:31.709 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:31.710 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:31.710 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:39:31.710 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:39:31.710 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:31.710 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:31.710 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:31.710 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:31.710 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:31.711 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:31.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.716+0000 7fb0d4243640 1 librados: starting msgr at 2024-03-29T10:39:31.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.716+0000 7fb0d4243640 1 librados: starting objecter 2024-03-29T10:39:31.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.716+0000 7fb0d4243640 1 Processor -- start 2024-03-29T10:39:31.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.716+0000 7fb0d4243640 1 -- start start 2024-03-29T10:39:31.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.716+0000 7fb0d4243640 1 librados: setting wanted keys 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.716+0000 7fb0d4243640 1 librados: calling monclient init 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.716+0000 7fb0d4243640 10 monclient: init 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.717+0000 7fb0d4243640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.717+0000 7fb0d4243640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.717+0000 7fb0d4243640 10 monclient(hunting): picked mon.noname-b con 0x7fb0cc0ac5a0 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.717+0000 7fb0d4243640 10 monclient(hunting): picked mon.noname-a con 0x7fb0cc0b0770 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.717+0000 7fb0d4243640 10 monclient(hunting): picked mon.noname-c con 0x7fb0cc14a700 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.717+0000 7fb0d4243640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc1dc4b0 con 0x7fb0cc0b0770 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.717+0000 7fb0d4243640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc1239d0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.717+0000 7fb0d4243640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc123b60 con 0x7fb0cc14a700 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.717+0000 7fb0d4243640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:31.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.717+0000 7fb0d4243640 10 monclient(hunting): authenticate will time out at 836.958747s 2024-03-29T10:39:31.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0d27e0640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fb0cc14a700 0x7fb0cc1dbda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:51402/0 (socket says 172.21.15.114:51402) 2024-03-29T10:39:31.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0d27e0640 1 -- 172.21.15.114:0/2116814495 learned_addr learned my addr 172.21.15.114:0/2116814495 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:31.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 10 client.?.objecter ms_handle_connect 0x7fb0cc14a700 2024-03-29T10:39:31.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:31.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 10 client.?.objecter ms_handle_connect 0x7fb0cc0ac5a0 2024-03-29T10:39:31.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:31.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881923770 0 0) 0x7fb0cc1239d0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:31.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:31.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:31.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:39:31.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b0004ee0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 10 client.?.objecter ms_handle_connect 0x7fb0cc0b0770 2024-03-29T10:39:31.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:31.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119533304 0 0) 0x7fb0cc123b60 con 0x7fb0cc14a700 2024-03-29T10:39:31.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.718+0000 7fb0c27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:31.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:31.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:31.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 10 monclient(hunting): my global_id is 4235 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0cc1239d0 con 0x7fb0cc14a700 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709652453 0 0) 0x7fb0cc1dc4b0 con 0x7fb0cc0b0770 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0cc123b60 con 0x7fb0cc0b0770 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 595367072 0 0) 0x7fb0b0004ee0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.719+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0cc1dc4b0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb0c4001c30 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:31.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient(hunting): dump: 2024-03-29T10:39:31.716 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:31.716 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:31.716 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.716 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.716 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:31.716 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:31.716 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:31.716 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:31.716 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:31.716 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:31.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:31.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1887037620 0 0) 0x7fb0cc123b60 con 0x7fb0cc0b0770 2024-03-29T10:39:31.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0b0004ee0 con 0x7fb0cc0b0770 2024-03-29T10:39:31.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb0bc001750 con 0x7fb0cc0b0770 2024-03-29T10:39:31.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:31.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:31.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient(hunting): dump: 2024-03-29T10:39:31.717 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:31.717 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:31.717 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.717 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2403727670 0 0) 0x7fb0cc1239d0 con 0x7fb0cc14a700 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0cc123b60 con 0x7fb0cc14a700 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb0c8001a30 con 0x7fb0cc14a700 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:31.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient(hunting): dump: 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4179899448 0 0) 0x7fb0cc1dc4b0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 >> v1:172.21.15.114:6790/0 conn(0x7fb0cc14a700 legacy=0x7fb0cc1dbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:31.719 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 >> v1:172.21.15.114:6789/0 conn(0x7fb0cc0b0770 legacy=0x7fb0cc125f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:31.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:31.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 monclient: found mon.b 2024-03-29T10:39:31.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:31.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:31.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc123cf0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:31.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:31.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.720+0000 7fb0c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:31.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0d4243640 5 monclient: authenticate success, global_id 4240 2024-03-29T10:39:31.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0d4243640 10 monclient: _renew_subs 2024-03-29T10:39:31.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0c40020c0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0d4243640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0d4243640 1 -- 172.21.15.114:0/2116814495 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0cc144ef0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0d4243640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb0c4003990 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0c3fff640 4 set_mon_vals no callback set 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0c27fc640 10 monclient: dump: 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:31.721 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0d4243640 10 monclient: _renew_subs 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0d4243640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0d4243640 1 -- 172.21.15.114:0/2116814495 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0cc145470 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.721+0000 7fb0d4243640 1 librados: init done 2024-03-29T10:39:31.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.723+0000 7fb0affff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:31.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.723+0000 7fb0affff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:31.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.723+0000 7fb0affff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:31.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.723+0000 7fb0affff640 1 -- 172.21.15.114:0/2116814495 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0cc0ab560 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.725+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fb0c4019960 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.726+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fb0c4083850 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.726+0000 7fb0d17de640 10 monclient: get_auth_request con 0x7fb0b006e150 auth_method 0 2024-03-29T10:39:31.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.726+0000 7fb0c27fc640 10 client.4240.objecter ms_dispatch 0x7fb0cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:31.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.726+0000 7fb0c27fc640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:31.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.726+0000 7fb0c27fc640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:31.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.726+0000 7fb0c27fc640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T10:39:31.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.726+0000 7fb0c27fc640 10 client.4240.objecter ms_handle_connect 0x7fb0b006e150 2024-03-29T10:39:31.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.733+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0c404ef60 con 0x7fb0cc0ac5a0 2024-03-29T10:39:31.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.733+0000 7fb0c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:31.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:31.733+0000 7fb0c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:32.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.103+0000 7fb0affff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:32.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.103+0000 7fb0affff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:32.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.103+0000 7fb0affff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:32.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.103+0000 7fb0affff640 1 -- 172.21.15.114:0/2116814495 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0cc1284b0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:32.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.104+0000 7fb0c27fc640 1 -- 172.21.15.114:0/2116814495 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fb0cc1284b0 con 0x7fb0cc0ac5a0 2024-03-29T10:39:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.104+0000 7fb0c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.104+0000 7fb0c27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:32.089 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:39:32.089 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:32.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.108+0000 7fb0affff640 10 librados: watch_flush enter 2024-03-29T10:39:32.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.108+0000 7fb0affff640 10 librados: watch_flush exit 2024-03-29T10:39:32.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.108+0000 7fb0affff640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T10:39:32.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.108+0000 7fb0affff640 10 client.4240.objecter successfully canceled tick 2024-03-29T10:39:32.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.108+0000 7fb0affff640 1 -- 172.21.15.114:0/2116814495 >> v1:172.21.15.114:6816/775044307 conn(0x7fb0b006e150 legacy=0x7fb0b0070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:32.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.108+0000 7fb0affff640 10 monclient: shutdown 2024-03-29T10:39:32.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.108+0000 7fb0affff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:32.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.108+0000 7fb0affff640 1 -- 172.21.15.114:0/2116814495 >> v1:172.21.15.176:6789/0 conn(0x7fb0cc0ac5a0 legacy=0x7fb0cc126d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:32.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.108+0000 7fb0affff640 1 -- 172.21.15.114:0/2116814495 shutdown_connections 2024-03-29T10:39:32.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.108+0000 7fb0affff640 1 -- 172.21.15.114:0/2116814495 >> 172.21.15.114:0/2116814495 conn(0x7fb0cc06b660 msgr2=0x7fb0cc0712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:32.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.109+0000 7fb0affff640 1 -- 172.21.15.114:0/2116814495 shutdown_connections 2024-03-29T10:39:32.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.109+0000 7fb0affff640 1 -- 172.21.15.114:0/2116814495 wait complete. 2024-03-29T10:39:32.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:32.109+0000 7fb0affff640 1 librados: shutdown 2024-03-29T10:39:32.108 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:38.916 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:39.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.073+0000 7f9625a7b640 10 monclient: get_monmap_and_config 2024-03-29T10:39:39.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.073+0000 7f9625a7b640 10 monclient: build_initial_monmap 2024-03-29T10:39:39.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.073+0000 7f9625a7b640 10 monclient: monmap: 2024-03-29T10:39:39.057 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:39.057 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:39.057 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:39:39.073968+0000 2024-03-29T10:39:39.057 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:39:39.073968+0000 2024-03-29T10:39:39.058 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:39.058 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:39.058 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:39.058 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:39.058 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:39.058 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:39.059 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.074+0000 7f9625a7b640 1 Processor -- start 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.074+0000 7f9625a7b640 1 -- start start 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.074+0000 7f9625a7b640 10 monclient: init 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.075+0000 7f9625a7b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.075+0000 7f9625a7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.075+0000 7f9625a7b640 10 monclient(hunting): picked mon.noname-a con 0x7f96201457f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.075+0000 7f9625a7b640 10 monclient(hunting): picked mon.noname-c con 0x7f962013f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.075+0000 7f9625a7b640 10 monclient(hunting): picked mon.noname-b con 0x7f96200a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.075+0000 7f9625a7b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96200aaef0 con 0x7f96201457f0 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.075+0000 7f9625a7b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9620145bd0 con 0x7f962013f650 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.075+0000 7f9625a7b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9620145eb0 con 0x7f96200a9d70 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.075+0000 7f9625a7b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:39.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.075+0000 7f9625a7b640 10 monclient(hunting): authenticate will time out at 844.315784s 2024-03-29T10:39:39.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.078+0000 7f961effd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f962013f650 0x7f96200b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:40804/0 (socket says 172.21.15.114:40804) 2024-03-29T10:39:39.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.078+0000 7f961effd640 1 -- 172.21.15.114:0/1970608924 learned_addr learned my addr 172.21.15.114:0/1970608924 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:39.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 1 -- 172.21.15.114:0/1970608924 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311502498 0 0) 0x7f9620145eb0 con 0x7f96200a9d70 2024-03-29T10:39:39.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:39.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:39.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:39.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 10 monclient(hunting): my global_id is 4238 2024-03-29T10:39:39.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 1 -- 172.21.15.114:0/1970608924 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f960c003520 con 0x7f96200a9d70 2024-03-29T10:39:39.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 1 -- 172.21.15.114:0/1970608924 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172890588 0 0) 0x7f9620145bd0 con 0x7f962013f650 2024-03-29T10:39:39.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:39.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:39.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:39.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:39:39.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.079+0000 7f961e7fc640 1 -- 172.21.15.114:0/1970608924 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9620145eb0 con 0x7f962013f650 2024-03-29T10:39:39.064 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.080+0000 7f961e7fc640 1 -- 172.21.15.114:0/1970608924 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1780268172 0 0) 0x7f960c003520 con 0x7f96200a9d70 2024-03-29T10:39:39.064 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.080+0000 7f961e7fc640 1 -- 172.21.15.114:0/1970608924 >> v1:172.21.15.176:6789/0 conn(0x7f962013f650 legacy=0x7f96200b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:39.064 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.080+0000 7f961e7fc640 1 -- 172.21.15.114:0/1970608924 >> v1:172.21.15.114:6789/0 conn(0x7f96201457f0 legacy=0x7f962013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:39.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.081+0000 7f961e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:39.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.081+0000 7f961e7fc640 1 monclient: found mon.noname-b 2024-03-29T10:39:39.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.081+0000 7f961e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:39.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:39:39.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 1 -- 172.21.15.114:0/1970608924 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9620146340 con 0x7f96200a9d70 2024-03-29T10:39:39.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:39.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:39.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:39.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 1 -- 172.21.15.114:0/1970608924 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9618001a10 con 0x7f96200a9d70 2024-03-29T10:39:39.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f9625a7b640 5 monclient: authenticate success, global_id 4238 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f9625a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 10 monclient: dump: 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:39.067 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:39.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f9625a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:39.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 1 -- 172.21.15.114:0/1970608924 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9618001c30 con 0x7f96200a9d70 2024-03-29T10:39:39.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:39.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f9625a7b640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:39.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f9625a7b640 4 set_mon_vals no callback set 2024-03-29T10:39:39.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 1 -- 172.21.15.114:0/1970608924 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9618001fc0 con 0x7f96200a9d70 2024-03-29T10:39:39.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.082+0000 7f961e7fc640 10 monclient: dump: 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.083+0000 7f9625a7b640 10 monclient: shutdown 2024-03-29T10:39:39.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.083+0000 7f9625a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:39.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.083+0000 7f9625a7b640 1 -- 172.21.15.114:0/1970608924 >> v1:172.21.15.114:6790/0 conn(0x7f96200a9d70 legacy=0x7f962014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:39.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.083+0000 7f9625a7b640 1 -- 172.21.15.114:0/1970608924 shutdown_connections 2024-03-29T10:39:39.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.083+0000 7f9625a7b640 1 -- 172.21.15.114:0/1970608924 >> 172.21.15.114:0/1970608924 conn(0x7f9620066a00 msgr2=0x7f9620066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:39.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.086+0000 7f9625a7b640 1 -- 172.21.15.114:0/1970608924 shutdown_connections 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.087+0000 7f9625a7b640 1 -- 172.21.15.114:0/1970608924 wait complete. 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.088+0000 7f9625a7b640 10 monclient: build_initial_monmap 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.088+0000 7f9625a7b640 10 monclient: monmap: 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:39.072 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:39.073 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:39.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.088+0000 7f9625a7b640 1 librados: starting msgr at 2024-03-29T10:39:39.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.088+0000 7f9625a7b640 1 librados: starting objecter 2024-03-29T10:39:39.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.088+0000 7f9625a7b640 1 Processor -- start 2024-03-29T10:39:39.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.089+0000 7f9625a7b640 1 -- start start 2024-03-29T10:39:39.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.089+0000 7f9625a7b640 1 librados: setting wanted keys 2024-03-29T10:39:39.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.089+0000 7f9625a7b640 1 librados: calling monclient init 2024-03-29T10:39:39.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.089+0000 7f9625a7b640 10 monclient: init 2024-03-29T10:39:39.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.089+0000 7f9625a7b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:39.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.089+0000 7f9625a7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:39.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.090+0000 7f9625a7b640 10 monclient(hunting): picked mon.noname-b con 0x7f96200a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:39.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.090+0000 7f9625a7b640 10 monclient(hunting): picked mon.noname-a con 0x7f962013f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:39.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.090+0000 7f9625a7b640 10 monclient(hunting): picked mon.noname-c con 0x7f96201457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:39.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.090+0000 7f9625a7b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96200b1670 con 0x7f962013f650 2024-03-29T10:39:39.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.090+0000 7f9625a7b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9620071b10 con 0x7f96200a9d70 2024-03-29T10:39:39.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.090+0000 7f9625a7b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962006f4b0 con 0x7f96201457f0 2024-03-29T10:39:39.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.090+0000 7f9625a7b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:39.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.090+0000 7f9625a7b640 10 monclient(hunting): authenticate will time out at 844.330076s 2024-03-29T10:39:39.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.092+0000 7f961f7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f96200a9d70 0x7f9620141d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:41254/0 (socket says 172.21.15.114:41254) 2024-03-29T10:39:39.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.092+0000 7f961f7fe640 1 -- 172.21.15.114:0/3447458399 learned_addr learned my addr 172.21.15.114:0/3447458399 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:39.082 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.094+0000 7f9617fff640 10 client.?.objecter ms_handle_connect 0x7f96200a9d70 2024-03-29T10:39:39.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.095+0000 7f9617fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:39.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.095+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251479378 0 0) 0x7f9620071b10 con 0x7f96200a9d70 2024-03-29T10:39:39.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.095+0000 7f9617fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:39.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.095+0000 7f9617fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:39.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.095+0000 7f9617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:39.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.095+0000 7f9617fff640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:39:39.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.095+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9600004eb0 con 0x7f96200a9d70 2024-03-29T10:39:39.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.095+0000 7f9617fff640 10 client.?.objecter ms_handle_connect 0x7f962013f650 2024-03-29T10:39:39.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.095+0000 7f9617fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.095+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896763234 0 0) 0x7f96200b1670 con 0x7f962013f650 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9620071b10 con 0x7f962013f650 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 client.?.objecter ms_handle_connect 0x7f96201457f0 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703200253 0 0) 0x7f962006f4b0 con 0x7f96201457f0 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:39.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:39.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:39:39.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96200b1670 con 0x7f96201457f0 2024-03-29T10:39:39.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3968995634 0 0) 0x7f9620071b10 con 0x7f962013f650 2024-03-29T10:39:39.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f962006f4b0 con 0x7f962013f650 2024-03-29T10:39:39.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9610001820 con 0x7f962013f650 2024-03-29T10:39:39.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:39.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:39.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): dump: 2024-03-29T10:39:39.085 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:39.086 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:39.086 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:39.086 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:39.086 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:39.086 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:39.086 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:39.086 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:39.086 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:39.087 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:39.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:39.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2755305422 0 0) 0x7f96200b1670 con 0x7f96201457f0 2024-03-29T10:39:39.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.096+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9620071b10 con 0x7f96201457f0 2024-03-29T10:39:39.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9618001950 con 0x7f96201457f0 2024-03-29T10:39:39.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:39.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:39.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 10 monclient(hunting): dump: 2024-03-29T10:39:39.087 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:39.087 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:39.087 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2422309019 0 0) 0x7f962006f4b0 con 0x7f962013f650 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 >> v1:172.21.15.114:6790/0 conn(0x7f96201457f0 legacy=0x7f96200700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 >> v1:172.21.15.176:6789/0 conn(0x7f96200a9d70 legacy=0x7f9620141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 1 monclient: found mon.a 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:39.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96201dc3b0 con 0x7f962013f650 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9617fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9625a7b640 5 monclient: authenticate success, global_id 4251 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9625a7b640 10 monclient: _renew_subs 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9625a7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9625a7b640 1 -- 172.21.15.114:0/3447458399 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962013a780 con 0x7f962013f650 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9625a7b640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9625a7b640 10 monclient: _renew_subs 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9625a7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9625a7b640 1 -- 172.21.15.114:0/3447458399 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9620071f30 con 0x7f962013f650 2024-03-29T10:39:39.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.097+0000 7f9625a7b640 1 librados: init done 2024-03-29T10:39:39.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.100+0000 7f9625a7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:39.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.100+0000 7f9625a7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:39.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.100+0000 7f9625a7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:39.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.100+0000 7f9625a7b640 1 -- 172.21.15.114:0/3447458399 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96200b1440 con 0x7f962013f650 2024-03-29T10:39:39.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.101+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9610001d10 con 0x7f962013f650 2024-03-29T10:39:39.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.101+0000 7f9617fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:39.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.101+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f96100047e0 con 0x7f962013f650 2024-03-29T10:39:39.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.101+0000 7f9617fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:39.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.101+0000 7f9617fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:39.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.101+0000 7f9617fff640 10 monclient: dump: 2024-03-29T10:39:39.092 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.101+0000 7f961dffb640 4 set_mon_vals no callback set 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.102+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f961001a7b0 con 0x7f962013f650 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.102+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f96100846f0 con 0x7f962013f650 2024-03-29T10:39:39.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.102+0000 7f9617fff640 10 client.4251.objecter ms_dispatch 0x7f9620000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:39.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.102+0000 7f9617fff640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:39.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.102+0000 7f9617fff640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:39.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.103+0000 7f9617fff640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T10:39:39.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.104+0000 7f961f7fe640 10 monclient: get_auth_request con 0x7f960006dfd0 auth_method 0 2024-03-29T10:39:39.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.105+0000 7f9617fff640 10 client.4251.objecter ms_handle_connect 0x7f960006dfd0 2024-03-29T10:39:39.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.109+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f961004fe00 con 0x7f962013f650 2024-03-29T10:39:39.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.109+0000 7f9617fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:39.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.109+0000 7f9617fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:39.423 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.439+0000 7f9625a7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:39.423 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.439+0000 7f9625a7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:39.424 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.439+0000 7f9625a7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:39.424 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.439+0000 7f9625a7b640 1 -- 172.21.15.114:0/3447458399 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96200b1820 con 0x7f962013f650 2024-03-29T10:39:39.424 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.440+0000 7f9617fff640 1 -- 172.21.15.114:0/3447458399 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f96200b1820 con 0x7f962013f650 2024-03-29T10:39:39.424 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.441+0000 7f9617fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:39.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.441+0000 7f9617fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:39.426 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:39:39.426 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:39.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.444+0000 7f9625a7b640 10 librados: watch_flush enter 2024-03-29T10:39:39.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.444+0000 7f9625a7b640 10 librados: watch_flush exit 2024-03-29T10:39:39.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.444+0000 7f9625a7b640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T10:39:39.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.444+0000 7f9625a7b640 10 client.4251.objecter successfully canceled tick 2024-03-29T10:39:39.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.445+0000 7f9625a7b640 1 -- 172.21.15.114:0/3447458399 >> v1:172.21.15.114:6816/775044307 conn(0x7f960006dfd0 legacy=0x7f9600070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:39.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.445+0000 7f9625a7b640 10 monclient: shutdown 2024-03-29T10:39:39.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.445+0000 7f9625a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:39.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.445+0000 7f9625a7b640 1 -- 172.21.15.114:0/3447458399 >> v1:172.21.15.114:6789/0 conn(0x7f962013f650 legacy=0x7f96201dbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:39.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.447+0000 7f9625a7b640 1 -- 172.21.15.114:0/3447458399 shutdown_connections 2024-03-29T10:39:39.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.447+0000 7f9625a7b640 1 -- 172.21.15.114:0/3447458399 >> 172.21.15.114:0/3447458399 conn(0x7f9620066a00 msgr2=0x7f9620066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:39.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.448+0000 7f9625a7b640 1 -- 172.21.15.114:0/3447458399 shutdown_connections 2024-03-29T10:39:39.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.448+0000 7f9625a7b640 1 -- 172.21.15.114:0/3447458399 wait complete. 2024-03-29T10:39:39.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:39.449+0000 7f9625a7b640 1 librados: shutdown 2024-03-29T10:39:39.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:46.258 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:46.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.429+0000 7f163a048640 10 monclient: get_monmap_and_config 2024-03-29T10:39:46.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.429+0000 7f163a048640 10 monclient: build_initial_monmap 2024-03-29T10:39:46.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.429+0000 7f163a048640 10 monclient: monmap: 2024-03-29T10:39:46.413 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:46.413 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:46.413 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:39:46.429646+0000 2024-03-29T10:39:46.413 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:39:46.429646+0000 2024-03-29T10:39:46.413 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:46.413 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:46.413 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:46.413 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:46.413 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:46.413 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:46.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.432+0000 7f163a048640 1 Processor -- start 2024-03-29T10:39:46.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.432+0000 7f163a048640 1 -- start start 2024-03-29T10:39:46.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.432+0000 7f163a048640 10 monclient: init 2024-03-29T10:39:46.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.433+0000 7f163a048640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:46.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.433+0000 7f163a048640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:46.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.433+0000 7f163a048640 10 monclient(hunting): picked mon.noname-c con 0x7f162c006e30 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:46.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.433+0000 7f163a048640 10 monclient(hunting): picked mon.noname-b con 0x7f162c035e00 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:46.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.433+0000 7f163a048640 10 monclient(hunting): picked mon.noname-a con 0x7f162c0c9670 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:46.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.433+0000 7f163a048640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c035970 con 0x7f162c0c9670 2024-03-29T10:39:46.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.433+0000 7f163a048640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c036280 con 0x7f162c006e30 2024-03-29T10:39:46.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.433+0000 7f163a048640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c035640 con 0x7f162c035e00 2024-03-29T10:39:46.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.433+0000 7f163a048640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:46.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.433+0000 7f163a048640 10 monclient(hunting): authenticate will time out at 851.673562s 2024-03-29T10:39:46.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.436+0000 7f1638845640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f162c035e00 0x7f162c0c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:51639/0 (socket says 172.21.15.114:51639) 2024-03-29T10:39:46.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.436+0000 7f1638845640 1 -- 172.21.15.114:0/3026981886 learned_addr learned my addr 172.21.15.114:0/3026981886 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:46.423 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.439+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020489217 0 0) 0x7f162c035970 con 0x7f162c0c9670 2024-03-29T10:39:46.423 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.439+0000 7f162bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:46.423 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.439+0000 7f162bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:46.423 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.439+0000 7f162bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:46.423 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.439+0000 7f162bfff640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:39:46.423 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.439+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1620003520 con 0x7f162c0c9670 2024-03-29T10:39:46.424 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2455128696 0 0) 0x7f1620003520 con 0x7f162c0c9670 2024-03-29T10:39:46.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 >> v1:172.21.15.114:6790/0 conn(0x7f162c035e00 legacy=0x7f162c0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:46.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 >> v1:172.21.15.176:6789/0 conn(0x7f162c006e30 legacy=0x7f162c0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:46.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:46.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 1 monclient: found mon.noname-a 2024-03-29T10:39:46.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:46.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:39:46.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c034e20 con 0x7f162c0c9670 2024-03-29T10:39:46.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 10 monclient: _finish_auth 0 2024-03-29T10:39:46.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:46.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:46.426 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f163a048640 5 monclient: authenticate success, global_id 4254 2024-03-29T10:39:46.426 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f163a048640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f163404f730 con 0x7f162c0c9670 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f162bfff640 10 monclient: dump: 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:46.427 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.440+0000 7f163a048640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.441+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16340008d0 con 0x7f162c0c9670 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.441+0000 7f162bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.441+0000 7f163a048640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.441+0000 7f163a048640 4 set_mon_vals no callback set 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.441+0000 7f1639847640 1 -- 172.21.15.114:0/3026981886 >> v1:172.21.15.114:6789/0 conn(0x7f162c0c9670 legacy=0x7f162c0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.441+0000 7f1639847640 1 -- 172.21.15.114:0/3026981886 >> v1:172.21.15.114:6789/0 conn(0x7f162c0c9670 legacy=0x7f162c0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.441+0000 7f1639847640 1 --1- 172.21.15.114:0/3026981886 >> v1:172.21.15.114:6789/0 conn(0x7f162c0c9670 0x7f162c0d7560 :-1 s=OPENED pgs=70472 cs=1 l=1).handle_message read tag failed 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.441+0000 7f1639847640 1 --1- 172.21.15.114:0/3026981886 >> v1:172.21.15.114:6789/0 conn(0x7f162c0c9670 0x7f162c0d7560 :-1 s=OPENED pgs=70472 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.441+0000 7f162bfff640 10 monclient: ms_handle_reset current mon v1:172.21.15.114:6789/0 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.441+0000 7f162bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.441+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 >> v1:172.21.15.114:6789/0 conn(0x7f162c0c9670 legacy=0x7f162c0d7560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f163a048640 10 monclient: shutdown 2024-03-29T10:39:46.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f162bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:46.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f162bfff640 10 monclient(hunting): picked mon.a con 0x7f1620003aa0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:46.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f162bfff640 10 monclient(hunting): picked mon.c con 0x7f1620007a10 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:46.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f162bfff640 10 monclient(hunting): picked mon.b con 0x7f162c006e30 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:46.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f162c036280 con 0x7f1620003aa0 2024-03-29T10:39:46.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f162c035640 con 0x7f162c006e30 2024-03-29T10:39:46.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f16200112f0 con 0x7f1620007a10 2024-03-29T10:39:46.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f162bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:46.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f162bfff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.114:6789/0 2024-03-29T10:39:46.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f163a048640 20 monclient(hunting): shutdown discarding 1 pending message(s) 2024-03-29T10:39:46.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.442+0000 7f163a048640 1 -- 172.21.15.114:0/3026981886 >> v1:172.21.15.114:6790/0 conn(0x7f1620007a10 legacy=0x7f1620009ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:46.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.445+0000 7f163a048640 1 -- 172.21.15.114:0/3026981886 >> v1:172.21.15.176:6789/0 conn(0x7f162c006e30 legacy=0x7f162000d910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:46.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.446+0000 7f162bfff640 1 -- 172.21.15.114:0/3026981886 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606305816 0 0) 0x7f162c036280 con 0x7f1620003aa0 2024-03-29T10:39:46.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.448+0000 7f163a048640 1 -- 172.21.15.114:0/3026981886 >> v1:172.21.15.114:6789/0 conn(0x7f1620003aa0 legacy=0x7f1620003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:46.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.448+0000 7f162bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-29T10:39:46.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.448+0000 7f163a048640 1 -- 172.21.15.114:0/3026981886 shutdown_connections 2024-03-29T10:39:46.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.448+0000 7f163a048640 1 -- 172.21.15.114:0/3026981886 >> 172.21.15.114:0/3026981886 conn(0x7f162c005c00 msgr2=0x7f162c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:46.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.449+0000 7f163a048640 1 -- 172.21.15.114:0/3026981886 shutdown_connections 2024-03-29T10:39:46.434 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.450+0000 7f163a048640 1 -- 172.21.15.114:0/3026981886 wait complete. 2024-03-29T10:39:46.434 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.450+0000 7f163a048640 10 monclient: build_initial_monmap 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.450+0000 7f163a048640 10 monclient: monmap: 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.451+0000 7f163a048640 1 librados: starting msgr at 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.451+0000 7f163a048640 1 librados: starting objecter 2024-03-29T10:39:46.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.451+0000 7f163a048640 1 Processor -- start 2024-03-29T10:39:46.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.452+0000 7f163a048640 1 -- start start 2024-03-29T10:39:46.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.452+0000 7f163a048640 1 librados: setting wanted keys 2024-03-29T10:39:46.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.452+0000 7f163a048640 1 librados: calling monclient init 2024-03-29T10:39:46.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.452+0000 7f163a048640 10 monclient: init 2024-03-29T10:39:46.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.452+0000 7f163a048640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:46.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.452+0000 7f163a048640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:46.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.452+0000 7f163a048640 10 monclient(hunting): picked mon.noname-a con 0x7f1620003aa0 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:46.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.452+0000 7f163a048640 10 monclient(hunting): picked mon.noname-c con 0x7f1620007a10 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:46.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.455+0000 7f163a048640 10 monclient(hunting): picked mon.noname-b con 0x7f162c006e30 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:46.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.455+0000 7f163a048640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c035640 con 0x7f1620003aa0 2024-03-29T10:39:46.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.455+0000 7f163a048640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16200112f0 con 0x7f162c006e30 2024-03-29T10:39:46.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.456+0000 7f163a048640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1620012570 con 0x7f1620007a10 2024-03-29T10:39:46.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.456+0000 7f163a048640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:46.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.456+0000 7f163a048640 10 monclient(hunting): authenticate will time out at 851.696080s 2024-03-29T10:39:46.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.456+0000 7f1639847640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f162c006e30 0x7f162c160520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:35753/0 (socket says 172.21.15.114:35753) 2024-03-29T10:39:46.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.456+0000 7f1639847640 1 -- 172.21.15.114:0/487775256 learned_addr learned my addr 172.21.15.114:0/487775256 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:46.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 client.?.objecter ms_handle_connect 0x7f162c006e30 2024-03-29T10:39:46.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:46.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 client.?.objecter ms_handle_connect 0x7f1620003aa0 2024-03-29T10:39:46.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:46.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575781996 0 0) 0x7f16200112f0 con 0x7f162c006e30 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1618004eb0 con 0x7f162c006e30 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669994060 0 0) 0x7f162c035640 con 0x7f1620003aa0 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.459+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16200112f0 con 0x7f1620003aa0 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 24738876 0 0) 0x7f16200112f0 con 0x7f1620003aa0 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f162c035640 con 0x7f1620003aa0 2024-03-29T10:39:46.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1630001550 con 0x7f1620003aa0 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:46.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 10 client.?.objecter ms_handle_connect 0x7f1620007a10 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3654217887 0 0) 0x7f162c035640 con 0x7f1620003aa0 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 >> v1:172.21.15.114:6790/0 conn(0x7f1620007a10 legacy=0x7f162c02d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 >> v1:172.21.15.176:6789/0 conn(0x7f162c006e30 legacy=0x7f162c160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 1 monclient: found mon.a 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c169d20 con 0x7f1620003aa0 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f1629ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f163a048640 5 monclient: authenticate success, global_id 4257 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f163a048640 10 monclient: _renew_subs 2024-03-29T10:39:46.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f163a048640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f163a048640 1 -- 172.21.15.114:0/487775256 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162c16a240 con 0x7f1620003aa0 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f163a048640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f163a048640 10 monclient: _renew_subs 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f163a048640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f163a048640 1 -- 172.21.15.114:0/487775256 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f162c02da40 con 0x7f1620003aa0 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.460+0000 7f163a048640 1 librados: init done 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.461+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1630001ba0 con 0x7f1620003aa0 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.461+0000 7f1629ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.461+0000 7f162affd640 4 set_mon_vals no callback set 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.462+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1630001550 con 0x7f1620003aa0 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.462+0000 7f1629ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.462+0000 7f1629ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.462+0000 7f1629ffb640 10 monclient: dump: 2024-03-29T10:39:46.450 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.462+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f16300195b0 con 0x7f1620003aa0 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.463+0000 7f163a048640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:46.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.463+0000 7f163a048640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:46.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.463+0000 7f163a048640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:46.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.463+0000 7f163a048640 1 -- 172.21.15.114:0/487775256 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1614005080 con 0x7f1620003aa0 2024-03-29T10:39:46.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.463+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f16300835c0 con 0x7f1620003aa0 2024-03-29T10:39:46.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.463+0000 7f1629ffb640 10 client.4257.objecter ms_dispatch 0x7f1634000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:46.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.463+0000 7f1629ffb640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:46.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.463+0000 7f1629ffb640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:46.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.463+0000 7f1629ffb640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T10:39:46.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.465+0000 7f1638845640 10 monclient: get_auth_request con 0x7f161806df30 auth_method 0 2024-03-29T10:39:46.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.467+0000 7f1629ffb640 10 client.4257.objecter ms_handle_connect 0x7f161806df30 2024-03-29T10:39:46.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.472+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f163004ed50 con 0x7f1620003aa0 2024-03-29T10:39:46.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.472+0000 7f1629ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:46.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.472+0000 7f1629ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:46.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.800+0000 7f163a048640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:46.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.800+0000 7f163a048640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:46.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.800+0000 7f163a048640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:39:46.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.800+0000 7f163a048640 1 -- 172.21.15.114:0/487775256 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1614005330 con 0x7f1620003aa0 2024-03-29T10:39:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.801+0000 7f1629ffb640 1 -- 172.21.15.114:0/487775256 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f163004ed50 con 0x7f1620003aa0 2024-03-29T10:39:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.801+0000 7f1629ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.801+0000 7f1629ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:46.786 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:39:46.786 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:46.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.805+0000 7f163a048640 10 librados: watch_flush enter 2024-03-29T10:39:46.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.805+0000 7f163a048640 10 librados: watch_flush exit 2024-03-29T10:39:46.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.805+0000 7f163a048640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T10:39:46.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.805+0000 7f163a048640 10 client.4257.objecter successfully canceled tick 2024-03-29T10:39:46.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.805+0000 7f163a048640 1 -- 172.21.15.114:0/487775256 >> v1:172.21.15.114:6816/775044307 conn(0x7f161806df30 legacy=0x7f16180703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:46.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.806+0000 7f163a048640 10 monclient: shutdown 2024-03-29T10:39:46.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.806+0000 7f163a048640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:46.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.806+0000 7f163a048640 1 -- 172.21.15.114:0/487775256 >> v1:172.21.15.114:6789/0 conn(0x7f1620003aa0 legacy=0x7f162c169530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:46.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.806+0000 7f163a048640 1 -- 172.21.15.114:0/487775256 shutdown_connections 2024-03-29T10:39:46.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.806+0000 7f163a048640 1 -- 172.21.15.114:0/487775256 >> 172.21.15.114:0/487775256 conn(0x7f162c005c00 msgr2=0x7f162c0c7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:46.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.806+0000 7f163a048640 1 -- 172.21.15.114:0/487775256 shutdown_connections 2024-03-29T10:39:46.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.806+0000 7f163a048640 1 -- 172.21.15.114:0/487775256 wait complete. 2024-03-29T10:39:46.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:46.806+0000 7f163a048640 1 librados: shutdown 2024-03-29T10:39:46.808 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:53.615 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:53.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.777+0000 7f2c2b274640 10 monclient: get_monmap_and_config 2024-03-29T10:39:53.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.777+0000 7f2c2b274640 10 monclient: build_initial_monmap 2024-03-29T10:39:53.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.777+0000 7f2c2b274640 10 monclient: monmap: 2024-03-29T10:39:53.761 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:53.761 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:53.762 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:39:53.778421+0000 2024-03-29T10:39:53.762 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:39:53.778421+0000 2024-03-29T10:39:53.762 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:53.762 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:53.762 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:53.762 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:39:53.762 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:39:53.762 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:53.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.780+0000 7f2c2b274640 1 Processor -- start 2024-03-29T10:39:53.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.780+0000 7f2c2b274640 1 -- start start 2024-03-29T10:39:53.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.780+0000 7f2c2b274640 10 monclient: init 2024-03-29T10:39:53.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.781+0000 7f2c2b274640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:53.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.781+0000 7f2c2b274640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:53.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.781+0000 7f2c2b274640 10 monclient(hunting): picked mon.noname-a con 0x7f2c240b0500 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:53.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.781+0000 7f2c2b274640 10 monclient(hunting): picked mon.noname-c con 0x7f2c240b08e0 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:53.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.781+0000 7f2c2b274640 10 monclient(hunting): picked mon.noname-b con 0x7f2c240aeca0 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:53.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.781+0000 7f2c2b274640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c24140620 con 0x7f2c240b0500 2024-03-29T10:39:53.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.781+0000 7f2c2b274640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2413daf0 con 0x7f2c240b08e0 2024-03-29T10:39:53.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.781+0000 7f2c2b274640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c240ae180 con 0x7f2c240aeca0 2024-03-29T10:39:53.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.781+0000 7f2c2b274640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:53.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.781+0000 7f2c2b274640 10 monclient(hunting): authenticate will time out at 859.022355s 2024-03-29T10:39:53.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.785+0000 7f2c2880f640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f2c240b08e0 0x7f2c2413d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:57297/0 (socket says 172.21.15.114:57297) 2024-03-29T10:39:53.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.785+0000 7f2c2880f640 1 -- 172.21.15.114:0/124000984 learned_addr learned my addr 172.21.15.114:0/124000984 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:53.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.786+0000 7f2c17fff640 1 -- 172.21.15.114:0/124000984 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609312092 0 0) 0x7f2c2413daf0 con 0x7f2c240b08e0 2024-03-29T10:39:53.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.786+0000 7f2c17fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:53.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.786+0000 7f2c17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:53.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.786+0000 7f2c17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:53.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.786+0000 7f2c17fff640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:39:53.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.786+0000 7f2c17fff640 1 -- 172.21.15.114:0/124000984 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c10003520 con 0x7f2c240b08e0 2024-03-29T10:39:53.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.786+0000 7f2c17fff640 1 -- 172.21.15.114:0/124000984 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 420828257 0 0) 0x7f2c10003520 con 0x7f2c240b08e0 2024-03-29T10:39:53.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.786+0000 7f2c17fff640 1 -- 172.21.15.114:0/124000984 >> v1:172.21.15.114:6790/0 conn(0x7f2c240aeca0 legacy=0x7f2c2414b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:53.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 1 -- 172.21.15.114:0/124000984 >> v1:172.21.15.114:6789/0 conn(0x7f2c240b0500 legacy=0x7f2c2413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:53.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:53.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 1 monclient: found mon.noname-c 2024-03-29T10:39:53.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:53.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:39:53.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 1 -- 172.21.15.114:0/124000984 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c240accf0 con 0x7f2c240b08e0 2024-03-29T10:39:53.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:53.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:53.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c2b274640 5 monclient: authenticate success, global_id 4252 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c2b274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 1 -- 172.21.15.114:0/124000984 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2c20001690 con 0x7f2c240b08e0 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c17fff640 10 monclient: dump: 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:53.775 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.788+0000 7f2c2b274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.789+0000 7f2c17fff640 1 -- 172.21.15.114:0/124000984 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c20001860 con 0x7f2c240b08e0 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.789+0000 7f2c17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.789+0000 7f2c2b274640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.789+0000 7f2c2b274640 4 set_mon_vals no callback set 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.789+0000 7f2c17fff640 1 -- 172.21.15.114:0/124000984 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2c20001cb0 con 0x7f2c240b08e0 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.789+0000 7f2c17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.789+0000 7f2c17fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.789+0000 7f2c17fff640 10 monclient: dump: 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:53.776 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.790+0000 7f2c2b274640 10 monclient: shutdown 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.790+0000 7f2c2b274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.790+0000 7f2c2b274640 1 -- 172.21.15.114:0/124000984 >> v1:172.21.15.176:6789/0 conn(0x7f2c240b08e0 legacy=0x7f2c2413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.790+0000 7f2c2b274640 1 -- 172.21.15.114:0/124000984 shutdown_connections 2024-03-29T10:39:53.777 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.790+0000 7f2c2b274640 1 -- 172.21.15.114:0/124000984 >> 172.21.15.114:0/124000984 conn(0x7f2c2406b660 msgr2=0x7f2c2406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:53.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.791+0000 7f2c2b274640 1 -- 172.21.15.114:0/124000984 shutdown_connections 2024-03-29T10:39:53.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.792+0000 7f2c2b274640 1 -- 172.21.15.114:0/124000984 wait complete. 2024-03-29T10:39:53.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.793+0000 7f2c2b274640 10 monclient: build_initial_monmap 2024-03-29T10:39:53.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.793+0000 7f2c2b274640 10 monclient: monmap: 2024-03-29T10:39:53.779 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:39:53.779 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:53.779 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:39:53.779 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:39:53.779 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.794+0000 7f2c2b274640 1 librados: starting msgr at 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.794+0000 7f2c2b274640 1 librados: starting objecter 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.794+0000 7f2c2b274640 1 Processor -- start 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.794+0000 7f2c2b274640 1 -- start start 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.794+0000 7f2c2b274640 1 librados: setting wanted keys 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.794+0000 7f2c2b274640 1 librados: calling monclient init 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.794+0000 7f2c2b274640 10 monclient: init 2024-03-29T10:39:53.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.795+0000 7f2c2b274640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:53.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.795+0000 7f2c2b274640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:53.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.795+0000 7f2c2b274640 10 monclient(hunting): picked mon.noname-b con 0x7f2c240aeca0 addr v1:172.21.15.176:6789/0 2024-03-29T10:39:53.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.795+0000 7f2c2b274640 10 monclient(hunting): picked mon.noname-a con 0x7f2c240b0500 addr v1:172.21.15.114:6789/0 2024-03-29T10:39:53.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.795+0000 7f2c2b274640 10 monclient(hunting): picked mon.noname-c con 0x7f2c241c62b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:39:53.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.795+0000 7f2c2b274640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c241c7080 con 0x7f2c240b0500 2024-03-29T10:39:53.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.795+0000 7f2c2b274640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c241c7ed0 con 0x7f2c240aeca0 2024-03-29T10:39:53.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.795+0000 7f2c2b274640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c241c6d50 con 0x7f2c241c62b0 2024-03-29T10:39:53.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.795+0000 7f2c2b274640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:53.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.795+0000 7f2c2b274640 10 monclient(hunting): authenticate will time out at 859.036417s 2024-03-29T10:39:53.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.800+0000 7f2c2880f640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2c240b0500 0x7f2c2406eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39495/0 (socket says 172.21.15.114:39495) 2024-03-29T10:39:53.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.800+0000 7f2c2880f640 1 -- 172.21.15.114:0/3628806177 learned_addr learned my addr 172.21.15.114:0/3628806177 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:39:53.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.801+0000 7f2c15ffb640 10 client.?.objecter ms_handle_connect 0x7f2c240b0500 2024-03-29T10:39:53.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.801+0000 7f2c15ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:53.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 10 client.?.objecter ms_handle_connect 0x7f2c240aeca0 2024-03-29T10:39:53.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250205410 0 0) 0x7f2c241c7080 con 0x7f2c240b0500 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c08004eb0 con 0x7f2c240b0500 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627552342 0 0) 0x7f2c241c7ed0 con 0x7f2c240aeca0 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:39:53.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.802+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c241c7080 con 0x7f2c240aeca0 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 514884759 0 0) 0x7f2c08004eb0 con 0x7f2c240b0500 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c241c7ed0 con 0x7f2c240b0500 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2c20001b70 con 0x7f2c240b0500 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:53.791 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3486732159 0 0) 0x7f2c241c7080 con 0x7f2c240aeca0 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c08004eb0 con 0x7f2c240aeca0 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2c1c001c30 con 0x7f2c240aeca0 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.803+0000 7f2c15ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:53.792 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:53.793 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:53.793 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:53.793 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:53.793 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:53.793 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:53.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.804+0000 7f2c15ffb640 10 client.?.objecter ms_handle_connect 0x7f2c241c62b0 2024-03-29T10:39:53.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.804+0000 7f2c15ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:53.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.805+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2218396237 0 0) 0x7f2c08004eb0 con 0x7f2c240aeca0 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.805+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 >> v1:172.21.15.114:6790/0 conn(0x7f2c241c62b0 legacy=0x7f2c241e60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.805+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 >> v1:172.21.15.114:6789/0 conn(0x7f2c240b0500 legacy=0x7f2c2406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.805+0000 7f2c15ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.805+0000 7f2c15ffb640 1 monclient: found mon.b 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.805+0000 7f2c15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.805+0000 7f2c15ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.805+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c241c8140 con 0x7f2c240aeca0 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.805+0000 7f2c15ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.805+0000 7f2c15ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.805+0000 7f2c15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.806+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c1c003990 con 0x7f2c240aeca0 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.806+0000 7f2c15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:53.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.806+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2c1c003ca0 con 0x7f2c240aeca0 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.806+0000 7f2c15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.806+0000 7f2c15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.806+0000 7f2c15ffb640 10 monclient: dump: 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:39:53.798 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.806+0000 7f2c2b274640 5 monclient: authenticate success, global_id 4255 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.806+0000 7f2c2b274640 10 monclient: _renew_subs 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.806+0000 7f2c2b274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.806+0000 7f2c2b274640 1 -- 172.21.15.114:0/3628806177 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c241c72a0 con 0x7f2c240aeca0 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.807+0000 7f2c2b274640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.807+0000 7f2c2b274640 10 monclient: _renew_subs 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.807+0000 7f2c2b274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.807+0000 7f2c2b274640 1 -- 172.21.15.114:0/3628806177 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c241e77f0 con 0x7f2c240aeca0 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.807+0000 7f2c2b274640 1 librados: init done 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.806+0000 7f2c177fe640 4 set_mon_vals no callback set 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.808+0000 7f2c2b274640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.808+0000 7f2c2b274640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.808+0000 7f2c2b274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:53.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.808+0000 7f2c2b274640 1 -- 172.21.15.114:0/3628806177 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c240ad0c0 con 0x7f2c240aeca0 2024-03-29T10:39:53.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.810+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f2c1c019c70 con 0x7f2c240aeca0 2024-03-29T10:39:53.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.810+0000 7f2c2880f640 10 monclient: get_auth_request con 0x7f2c0806e0f0 auth_method 0 2024-03-29T10:39:53.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.811+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f2c1c053820 con 0x7f2c240aeca0 2024-03-29T10:39:53.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.811+0000 7f2c15ffb640 10 client.4255.objecter ms_dispatch 0x7f2c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:53.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.811+0000 7f2c15ffb640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:53.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.811+0000 7f2c15ffb640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:53.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.811+0000 7f2c15ffb640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-29T10:39:53.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.811+0000 7f2c15ffb640 10 client.4255.objecter ms_handle_connect 0x7f2c0806e0f0 2024-03-29T10:39:53.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.819+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c1c04fac0 con 0x7f2c240aeca0 2024-03-29T10:39:53.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.819+0000 7f2c15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:53.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:53.819+0000 7f2c15ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:54.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.142+0000 7f2c2b274640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:54.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.142+0000 7f2c2b274640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:54.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.142+0000 7f2c2b274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:39:54.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.142+0000 7f2c2b274640 1 -- 172.21.15.114:0/3628806177 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c24075f10 con 0x7f2c240aeca0 2024-03-29T10:39:54.129 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.145+0000 7f2c15ffb640 1 -- 172.21.15.114:0/3628806177 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f2c1c04fc60 con 0x7f2c240aeca0 2024-03-29T10:39:54.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.146+0000 7f2c15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:54.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.146+0000 7f2c15ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:54.130 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:39:54.130 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:54.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 10 librados: watch_flush enter 2024-03-29T10:39:54.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 10 librados: watch_flush exit 2024-03-29T10:39:54.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-29T10:39:54.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 10 client.4255.objecter successfully canceled tick 2024-03-29T10:39:54.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 1 -- 172.21.15.114:0/3628806177 >> v1:172.21.15.114:6816/775044307 conn(0x7f2c0806e0f0 legacy=0x7f2c080705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:54.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 10 monclient: shutdown 2024-03-29T10:39:54.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:54.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 1 -- 172.21.15.114:0/3628806177 >> v1:172.21.15.176:6789/0 conn(0x7f2c240aeca0 legacy=0x7f2c2406e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:54.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 1 -- 172.21.15.114:0/3628806177 shutdown_connections 2024-03-29T10:39:54.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 1 -- 172.21.15.114:0/3628806177 >> 172.21.15.114:0/3628806177 conn(0x7f2c2406b660 msgr2=0x7f2c240a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:54.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 1 -- 172.21.15.114:0/3628806177 shutdown_connections 2024-03-29T10:39:54.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.149+0000 7f2c2b274640 1 -- 172.21.15.114:0/3628806177 wait complete. 2024-03-29T10:39:54.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:39:54.150+0000 7f2c2b274640 1 librados: shutdown 2024-03-29T10:39:54.156 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:00.963 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:01.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.129+0000 7fd4cfe4e640 10 monclient: get_monmap_and_config 2024-03-29T10:40:01.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.129+0000 7fd4cfe4e640 10 monclient: build_initial_monmap 2024-03-29T10:40:01.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.129+0000 7fd4cfe4e640 10 monclient: monmap: 2024-03-29T10:40:01.114 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:01.115 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:01.115 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:40:01.130893+0000 2024-03-29T10:40:01.115 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:40:01.130893+0000 2024-03-29T10:40:01.115 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:01.115 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:01.115 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:01.115 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:40:01.115 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:40:01.115 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:01.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.130+0000 7fd4cfe4e640 1 Processor -- start 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 1 -- start start 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 10 monclient: init 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 10 monclient(hunting): picked mon.noname-c con 0x7fd4c0006e30 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 10 monclient(hunting): picked mon.noname-b con 0x7fd4c0035e00 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 10 monclient(hunting): picked mon.noname-a con 0x7fd4c00c9670 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c0035970 con 0x7fd4c00c9670 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c0036280 con 0x7fd4c0006e30 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c0035640 con 0x7fd4c0035e00 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:01.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.131+0000 7fd4cfe4e640 10 monclient(hunting): authenticate will time out at 866.372936s 2024-03-29T10:40:01.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.135+0000 7fd4cd3e9640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fd4c0035e00 0x7fd4c00c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39973/0 (socket says 172.21.15.114:39973) 2024-03-29T10:40:01.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.135+0000 7fd4cd3e9640 1 -- 172.21.15.114:0/3986035774 learned_addr learned my addr 172.21.15.114:0/3986035774 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:01.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.137+0000 7fd4ccbe8640 1 -- 172.21.15.114:0/3986035774 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225862002 0 0) 0x7fd4c0035640 con 0x7fd4c0035e00 2024-03-29T10:40:01.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.137+0000 7fd4ccbe8640 10 monclient(hunting): _negotiate 2024-03-29T10:40:01.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.137+0000 7fd4ccbe8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:01.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.137+0000 7fd4ccbe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:01.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.137+0000 7fd4ccbe8640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:40:01.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.137+0000 7fd4ccbe8640 1 -- 172.21.15.114:0/3986035774 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4b4003520 con 0x7fd4c0035e00 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 1 -- 172.21.15.114:0/3986035774 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188576010 0 0) 0x7fd4c0035970 con 0x7fd4c00c9670 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 10 monclient(hunting): _negotiate 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 1 -- 172.21.15.114:0/3986035774 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c0035640 con 0x7fd4c00c9670 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 1 -- 172.21.15.114:0/3986035774 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 834597662 0 0) 0x7fd4b4003520 con 0x7fd4c0035e00 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 1 -- 172.21.15.114:0/3986035774 >> v1:172.21.15.176:6789/0 conn(0x7fd4c0006e30 legacy=0x7fd4c00cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 1 -- 172.21.15.114:0/3986035774 >> v1:172.21.15.114:6789/0 conn(0x7fd4c00c9670 legacy=0x7fd4c00d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 10 monclient: _finish_hunting 0 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 1 monclient: found mon.noname-b 2024-03-29T10:40:01.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 1 -- 172.21.15.114:0/3986035774 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c0034e20 con 0x7fd4c0035e00 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 10 monclient: _finish_auth 0 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 10 monclient: _check_auth_tickets 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4cfe4e640 5 monclient: authenticate success, global_id 4247 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4cfe4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 1 -- 172.21.15.114:0/3986035774 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd4c804f730 con 0x7fd4c0035e00 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4ccbe8640 10 monclient: dump: 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:01.124 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:01.125 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:01.125 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:01.125 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:01.125 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:01.125 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:01.125 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:01.125 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:01.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.138+0000 7fd4cfe4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:01.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.139+0000 7fd4ccbe8640 1 -- 172.21.15.114:0/3986035774 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4c8075f10 con 0x7fd4c0035e00 2024-03-29T10:40:01.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.139+0000 7fd4ccbe8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:01.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.139+0000 7fd4cfe4e640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:01.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.139+0000 7fd4cfe4e640 4 set_mon_vals no callback set 2024-03-29T10:40:01.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.139+0000 7fd4ccbe8640 1 -- 172.21.15.114:0/3986035774 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd4c80758b0 con 0x7fd4c0035e00 2024-03-29T10:40:01.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.139+0000 7fd4ccbe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:01.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.139+0000 7fd4ccbe8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:01.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.139+0000 7fd4ccbe8640 10 monclient: dump: 2024-03-29T10:40:01.126 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:01.126 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.140+0000 7fd4cfe4e640 10 monclient: shutdown 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.140+0000 7fd4cfe4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.140+0000 7fd4cfe4e640 1 -- 172.21.15.114:0/3986035774 >> v1:172.21.15.114:6790/0 conn(0x7fd4c0035e00 legacy=0x7fd4c00c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.140+0000 7fd4cfe4e640 1 -- 172.21.15.114:0/3986035774 shutdown_connections 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.140+0000 7fd4cfe4e640 1 -- 172.21.15.114:0/3986035774 >> 172.21.15.114:0/3986035774 conn(0x7fd4c0005c00 msgr2=0x7fd4c0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.141+0000 7fd4cfe4e640 1 -- 172.21.15.114:0/3986035774 shutdown_connections 2024-03-29T10:40:01.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.141+0000 7fd4cfe4e640 1 -- 172.21.15.114:0/3986035774 wait complete. 2024-03-29T10:40:01.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.142+0000 7fd4cfe4e640 10 monclient: build_initial_monmap 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.142+0000 7fd4cfe4e640 10 monclient: monmap: 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.142+0000 7fd4cfe4e640 1 librados: starting msgr at 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.142+0000 7fd4cfe4e640 1 librados: starting objecter 2024-03-29T10:40:01.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.142+0000 7fd4cfe4e640 1 Processor -- start 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.142+0000 7fd4cfe4e640 1 -- start start 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.142+0000 7fd4cfe4e640 1 librados: setting wanted keys 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.142+0000 7fd4cfe4e640 1 librados: calling monclient init 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.142+0000 7fd4cfe4e640 10 monclient: init 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.143+0000 7fd4cfe4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.143+0000 7fd4cfe4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.143+0000 7fd4cfe4e640 10 monclient(hunting): picked mon.noname-a con 0x7fd4c0006e30 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.143+0000 7fd4cfe4e640 10 monclient(hunting): picked mon.noname-c con 0x7fd4c0035e00 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.143+0000 7fd4cfe4e640 10 monclient(hunting): picked mon.noname-b con 0x7fd4c00c9670 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.143+0000 7fd4cfe4e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c002e910 con 0x7fd4c0006e30 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.143+0000 7fd4cfe4e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c00cd560 con 0x7fd4c00c9670 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.143+0000 7fd4cfe4e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c00ce020 con 0x7fd4c0035e00 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.143+0000 7fd4cfe4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:01.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.143+0000 7fd4cfe4e640 10 monclient(hunting): authenticate will time out at 866.384534s 2024-03-29T10:40:01.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.148+0000 7fd4cd3e9640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fd4c0035e00 0x7fd4c00cd910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:40085/0 (socket says 172.21.15.114:40085) 2024-03-29T10:40:01.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.148+0000 7fd4cd3e9640 1 -- 172.21.15.114:0/583637451 learned_addr learned my addr 172.21.15.114:0/583637451 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:01.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.148+0000 7fd4be7fc640 10 client.?.objecter ms_handle_connect 0x7fd4c0035e00 2024-03-29T10:40:01.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.148+0000 7fd4be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:01.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231726994 0 0) 0x7fd4c00ce020 con 0x7fd4c0035e00 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ac004eb0 con 0x7fd4c0035e00 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4046205866 0 0) 0x7fd4ac004eb0 con 0x7fd4c0035e00 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4c00ce020 con 0x7fd4c0035e00 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd4c804f400 con 0x7fd4c0035e00 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 10 monclient(hunting): dump: 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:01.134 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:01.135 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:01.135 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:01.135 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:01.135 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:01.135 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:01.135 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:01.135 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:01.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.149+0000 7fd4be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:01.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.150+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 546365126 0 0) 0x7fd4c00ce020 con 0x7fd4c0035e00 2024-03-29T10:40:01.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.150+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 >> v1:172.21.15.176:6789/0 conn(0x7fd4c00c9670 legacy=0x7fd4c01741c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:40:01.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.150+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 >> v1:172.21.15.114:6789/0 conn(0x7fd4c0006e30 legacy=0x7fd4c002ec80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:01.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4be7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:01.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4be7fc640 1 monclient: found mon.c 2024-03-29T10:40:01.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:01.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4be7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c00ce1b0 con 0x7fd4c0035e00 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4be7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4be7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4cfe4e640 5 monclient: authenticate success, global_id 4250 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4cfe4e640 10 monclient: _renew_subs 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4cfe4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4cfe4e640 1 -- 172.21.15.114:0/583637451 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c002f500 con 0x7fd4c0035e00 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4cfe4e640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4cfe4e640 10 monclient: _renew_subs 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4cfe4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4cfe4e640 1 -- 172.21.15.114:0/583637451 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c0032d50 con 0x7fd4c0035e00 2024-03-29T10:40:01.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.152+0000 7fd4cfe4e640 1 librados: init done 2024-03-29T10:40:01.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.153+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4c804f730 con 0x7fd4c0035e00 2024-03-29T10:40:01.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.153+0000 7fd4be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:01.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.153+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd4c8067d90 con 0x7fd4c0035e00 2024-03-29T10:40:01.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.153+0000 7fd4be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:01.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.153+0000 7fd4be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:01.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.153+0000 7fd4be7fc640 10 monclient: dump: 2024-03-29T10:40:01.140 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:01.140 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:01.140 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:01.140 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:01.140 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:01.141 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:01.141 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:01.141 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:01.141 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:01.141 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:01.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.153+0000 7fd4bffff640 4 set_mon_vals no callback set 2024-03-29T10:40:01.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.154+0000 7fd4abfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:01.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.154+0000 7fd4abfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:01.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.154+0000 7fd4abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:01.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.154+0000 7fd4abfff640 1 -- 172.21.15.114:0/583637451 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c0007b90 con 0x7fd4c0035e00 2024-03-29T10:40:01.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.158+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fd4c8067ef0 con 0x7fd4c0035e00 2024-03-29T10:40:01.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.159+0000 7fd4cdbea640 10 monclient: get_auth_request con 0x7fd4ac06dbf0 auth_method 0 2024-03-29T10:40:01.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.163+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fd4c80ff6a0 con 0x7fd4c0035e00 2024-03-29T10:40:01.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.163+0000 7fd4be7fc640 10 client.4250.objecter ms_dispatch 0x7fd4c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:01.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.163+0000 7fd4be7fc640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:01.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.163+0000 7fd4be7fc640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:01.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.163+0000 7fd4be7fc640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T10:40:01.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.163+0000 7fd4be7fc640 10 client.4250.objecter ms_handle_connect 0x7fd4ac06dbf0 2024-03-29T10:40:01.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.164+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4c80caf80 con 0x7fd4c0035e00 2024-03-29T10:40:01.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.165+0000 7fd4be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:01.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.165+0000 7fd4be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:01.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.494+0000 7fd4abfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:01.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.494+0000 7fd4abfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:01.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.494+0000 7fd4abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:01.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.494+0000 7fd4abfff640 1 -- 172.21.15.114:0/583637451 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4c0005120 con 0x7fd4c0035e00 2024-03-29T10:40:01.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.498+0000 7fd4be7fc640 1 -- 172.21.15.114:0/583637451 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fd4c80cceb0 con 0x7fd4c0035e00 2024-03-29T10:40:01.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.498+0000 7fd4be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:01.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.498+0000 7fd4be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:01.483 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:40:01.483 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:01.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.501+0000 7fd4abfff640 10 librados: watch_flush enter 2024-03-29T10:40:01.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.503+0000 7fd4abfff640 10 librados: watch_flush exit 2024-03-29T10:40:01.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.503+0000 7fd4abfff640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T10:40:01.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.503+0000 7fd4abfff640 10 client.4250.objecter successfully canceled tick 2024-03-29T10:40:01.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.504+0000 7fd4abfff640 1 -- 172.21.15.114:0/583637451 >> v1:172.21.15.114:6816/775044307 conn(0x7fd4ac06dbf0 legacy=0x7fd4ac0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:01.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.504+0000 7fd4abfff640 10 monclient: shutdown 2024-03-29T10:40:01.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.504+0000 7fd4abfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:01.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.504+0000 7fd4abfff640 1 -- 172.21.15.114:0/583637451 >> v1:172.21.15.114:6790/0 conn(0x7fd4c0035e00 legacy=0x7fd4c00cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:01.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.505+0000 7fd4abfff640 1 -- 172.21.15.114:0/583637451 shutdown_connections 2024-03-29T10:40:01.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.505+0000 7fd4abfff640 1 -- 172.21.15.114:0/583637451 >> 172.21.15.114:0/583637451 conn(0x7fd4c0005c00 msgr2=0x7fd4c0005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:01.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.505+0000 7fd4abfff640 1 -- 172.21.15.114:0/583637451 shutdown_connections 2024-03-29T10:40:01.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.505+0000 7fd4abfff640 1 -- 172.21.15.114:0/583637451 wait complete. 2024-03-29T10:40:01.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:01.505+0000 7fd4abfff640 1 librados: shutdown 2024-03-29T10:40:01.507 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:08.314 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.482+0000 7ff697251640 10 monclient: get_monmap_and_config 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.482+0000 7ff697251640 10 monclient: build_initial_monmap 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.482+0000 7ff697251640 10 monclient: monmap: 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:40:08.484196+0000 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:40:08.484196+0000 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:40:08.467 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:40:08.468 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:08.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.484+0000 7ff697251640 1 Processor -- start 2024-03-29T10:40:08.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.484+0000 7ff697251640 1 -- start start 2024-03-29T10:40:08.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.484+0000 7ff697251640 10 monclient: init 2024-03-29T10:40:08.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.485+0000 7ff697251640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:08.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.485+0000 7ff697251640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:08.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.485+0000 7ff697251640 10 monclient(hunting): picked mon.noname-c con 0x7ff6901457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:08.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.485+0000 7ff697251640 10 monclient(hunting): picked mon.noname-b con 0x7ff69013f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:08.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.485+0000 7ff697251640 10 monclient(hunting): picked mon.noname-a con 0x7ff6900a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:08.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.485+0000 7ff697251640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6900aaef0 con 0x7ff6900a9d70 2024-03-29T10:40:08.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.485+0000 7ff697251640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff690145bd0 con 0x7ff6901457f0 2024-03-29T10:40:08.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.485+0000 7ff697251640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff690145eb0 con 0x7ff69013f650 2024-03-29T10:40:08.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.485+0000 7ff697251640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:08.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.485+0000 7ff697251640 10 monclient(hunting): authenticate will time out at 873.726685s 2024-03-29T10:40:08.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.489+0000 7ff6957ee640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff6900a9d70 0x7ff69014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35005/0 (socket says 172.21.15.114:35005) 2024-03-29T10:40:08.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.489+0000 7ff6957ee640 1 -- 172.21.15.114:0/2431800573 learned_addr learned my addr 172.21.15.114:0/2431800573 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:08.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.490+0000 7ff68f7fe640 1 -- 172.21.15.114:0/2431800573 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630080140 0 0) 0x7ff6900aaef0 con 0x7ff6900a9d70 2024-03-29T10:40:08.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.490+0000 7ff68f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:08.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.490+0000 7ff68f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:08.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.490+0000 7ff68f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:08.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.490+0000 7ff68f7fe640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:40:08.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.490+0000 7ff68f7fe640 1 -- 172.21.15.114:0/2431800573 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff678003520 con 0x7ff6900a9d70 2024-03-29T10:40:08.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 1 -- 172.21.15.114:0/2431800573 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106246018 0 0) 0x7ff690145eb0 con 0x7ff69013f650 2024-03-29T10:40:08.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:08.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:08.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:08.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:40:08.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 1 -- 172.21.15.114:0/2431800573 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6900aaef0 con 0x7ff69013f650 2024-03-29T10:40:08.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 1 -- 172.21.15.114:0/2431800573 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1871907652 0 0) 0x7ff6900aaef0 con 0x7ff69013f650 2024-03-29T10:40:08.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 1 -- 172.21.15.114:0/2431800573 >> v1:172.21.15.176:6789/0 conn(0x7ff6901457f0 legacy=0x7ff69013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:08.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 1 -- 172.21.15.114:0/2431800573 >> v1:172.21.15.114:6789/0 conn(0x7ff6900a9d70 legacy=0x7ff69014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:08.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:08.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 1 -- 172.21.15.114:0/2431800573 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff690146340 con 0x7ff69013f650 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 1 -- 172.21.15.114:0/2431800573 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff680001630 con 0x7ff69013f650 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient: dump: 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:08.479 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 1 -- 172.21.15.114:0/2431800573 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff680001880 con 0x7ff69013f650 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 1 -- 172.21.15.114:0/2431800573 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff680001630 con 0x7ff69013f650 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.492+0000 7ff68f7fe640 10 monclient: dump: 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:08.480 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:08.481 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:08.481 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:08.481 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:08.481 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:08.481 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:08.481 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:08.481 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:08.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.494+0000 7ff697251640 5 monclient: authenticate success, global_id 4253 2024-03-29T10:40:08.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.494+0000 7ff697251640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:08.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.494+0000 7ff697251640 4 set_mon_vals no callback set 2024-03-29T10:40:08.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.495+0000 7ff697251640 10 monclient: shutdown 2024-03-29T10:40:08.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.495+0000 7ff697251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:08.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.495+0000 7ff697251640 1 -- 172.21.15.114:0/2431800573 >> v1:172.21.15.114:6790/0 conn(0x7ff69013f650 legacy=0x7ff6900b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:08.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.498+0000 7ff697251640 1 -- 172.21.15.114:0/2431800573 shutdown_connections 2024-03-29T10:40:08.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.498+0000 7ff697251640 1 -- 172.21.15.114:0/2431800573 >> 172.21.15.114:0/2431800573 conn(0x7ff690066a00 msgr2=0x7ff690066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.502+0000 7ff697251640 1 -- 172.21.15.114:0/2431800573 shutdown_connections 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.503+0000 7ff697251640 1 -- 172.21.15.114:0/2431800573 wait complete. 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.503+0000 7ff697251640 10 monclient: build_initial_monmap 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.503+0000 7ff697251640 10 monclient: monmap: 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:40:08.489 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:08.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.504+0000 7ff697251640 1 librados: starting msgr at 2024-03-29T10:40:08.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.504+0000 7ff697251640 1 librados: starting objecter 2024-03-29T10:40:08.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.504+0000 7ff697251640 1 Processor -- start 2024-03-29T10:40:08.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.504+0000 7ff697251640 1 -- start start 2024-03-29T10:40:08.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.504+0000 7ff697251640 1 librados: setting wanted keys 2024-03-29T10:40:08.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.504+0000 7ff697251640 1 librados: calling monclient init 2024-03-29T10:40:08.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.504+0000 7ff697251640 10 monclient: init 2024-03-29T10:40:08.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.505+0000 7ff697251640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:08.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.505+0000 7ff697251640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:08.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.506+0000 7ff697251640 10 monclient(hunting): picked mon.noname-a con 0x7ff6900a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:08.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.506+0000 7ff697251640 10 monclient(hunting): picked mon.noname-c con 0x7ff69013f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:08.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.506+0000 7ff697251640 10 monclient(hunting): picked mon.noname-b con 0x7ff6901457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:08.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.506+0000 7ff697251640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff690142c60 con 0x7ff6900a9d70 2024-03-29T10:40:08.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.506+0000 7ff697251640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69006f8c0 con 0x7ff6901457f0 2024-03-29T10:40:08.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.506+0000 7ff697251640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69006fb30 con 0x7ff69013f650 2024-03-29T10:40:08.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.506+0000 7ff697251640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:08.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.506+0000 7ff697251640 10 monclient(hunting): authenticate will time out at 873.747086s 2024-03-29T10:40:08.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff6957ee640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ff6901457f0 0x7ff6901eec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:55027/0 (socket says 172.21.15.114:55027) 2024-03-29T10:40:08.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff6957ee640 1 -- 172.21.15.114:0/38518825 learned_addr learned my addr 172.21.15.114:0/38518825 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:08.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 client.?.objecter ms_handle_connect 0x7ff6901457f0 2024-03-29T10:40:08.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:08.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 client.?.objecter ms_handle_connect 0x7ff69013f650 2024-03-29T10:40:08.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:08.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322463771 0 0) 0x7ff69006f8c0 con 0x7ff6901457f0 2024-03-29T10:40:08.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c004eb0 con 0x7ff6901457f0 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328659411 0 0) 0x7ff69006fb30 con 0x7ff69013f650 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.510+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff69006f8c0 con 0x7ff69013f650 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.511+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2543497215 0 0) 0x7ff67c004eb0 con 0x7ff6901457f0 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.511+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff69006fb30 con 0x7ff6901457f0 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.511+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff688001cd0 con 0x7ff6901457f0 2024-03-29T10:40:08.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.511+0000 7ff68d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.511+0000 7ff68d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.511+0000 7ff68d7fa640 10 monclient(hunting): dump: 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:08.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.511+0000 7ff68d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:08.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.511+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3259321107 0 0) 0x7ff69006fb30 con 0x7ff6901457f0 2024-03-29T10:40:08.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.511+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 >> v1:172.21.15.114:6790/0 conn(0x7ff69013f650 legacy=0x7ff6901e78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:08.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.511+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 >> v1:172.21.15.114:6789/0 conn(0x7ff6900a9d70 legacy=0x7ff690141020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:08.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.512+0000 7ff68d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:40:08.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.512+0000 7ff68d7fa640 1 monclient: found mon.b 2024-03-29T10:40:08.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.512+0000 7ff68d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:08.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.512+0000 7ff68d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:08.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.512+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69006fe10 con 0x7ff6901457f0 2024-03-29T10:40:08.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff68d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:40:08.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff68d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:40:08.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff68d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:08.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff697251640 5 monclient: authenticate success, global_id 4258 2024-03-29T10:40:08.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff697251640 10 monclient: _renew_subs 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff697251640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff697251640 1 -- 172.21.15.114:0/38518825 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6901428f0 con 0x7ff6901457f0 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff697251640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff697251640 10 monclient: _renew_subs 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff697251640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff697251640 1 -- 172.21.15.114:0/38518825 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6901e7ff0 con 0x7ff6901457f0 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.513+0000 7ff697251640 1 librados: init done 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.514+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff688004b60 con 0x7ff6901457f0 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.514+0000 7ff68d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.514+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff688004eb0 con 0x7ff6901457f0 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.514+0000 7ff68effd640 4 set_mon_vals no callback set 2024-03-29T10:40:08.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.514+0000 7ff697251640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:08.501 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.515+0000 7ff697251640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:08.501 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.515+0000 7ff697251640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:08.501 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.515+0000 7ff697251640 1 -- 172.21.15.114:0/38518825 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff670005080 con 0x7ff6901457f0 2024-03-29T10:40:08.501 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.515+0000 7ff68d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:08.501 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.515+0000 7ff68d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:08.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.515+0000 7ff68d7fa640 10 monclient: dump: 2024-03-29T10:40:08.502 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:08.502 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:08.502 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:08.502 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:08.502 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:08.502 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:08.502 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:08.502 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:08.502 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:08.502 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:08.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.517+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7ff688005010 con 0x7ff6901457f0 2024-03-29T10:40:08.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.518+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7ff688084dc0 con 0x7ff6901457f0 2024-03-29T10:40:08.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.518+0000 7ff68d7fa640 10 client.4258.objecter ms_dispatch 0x7ff690000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:08.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.518+0000 7ff68d7fa640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:08.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.518+0000 7ff68d7fa640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:08.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.519+0000 7ff68d7fa640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-29T10:40:08.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.519+0000 7ff694fed640 10 monclient: get_auth_request con 0x7ff67c06de90 auth_method 0 2024-03-29T10:40:08.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.520+0000 7ff68d7fa640 10 client.4258.objecter ms_handle_connect 0x7ff67c06de90 2024-03-29T10:40:08.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.524+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6880506a0 con 0x7ff6901457f0 2024-03-29T10:40:08.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.524+0000 7ff68d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:08.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.524+0000 7ff68d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:08.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.841+0000 7ff697251640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:08.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.841+0000 7ff697251640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:08.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.841+0000 7ff697251640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:08.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.841+0000 7ff697251640 1 -- 172.21.15.114:0/38518825 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff670005330 con 0x7ff6901457f0 2024-03-29T10:40:08.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.844+0000 7ff68d7fa640 1 -- 172.21.15.114:0/38518825 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7ff6880525d0 con 0x7ff6901457f0 2024-03-29T10:40:08.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.844+0000 7ff68d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:08.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.844+0000 7ff68d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:08.830 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:40:08.830 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:08.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.850+0000 7ff697251640 10 librados: watch_flush enter 2024-03-29T10:40:08.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.850+0000 7ff697251640 10 librados: watch_flush exit 2024-03-29T10:40:08.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.850+0000 7ff697251640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-29T10:40:08.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.850+0000 7ff697251640 10 client.4258.objecter successfully canceled tick 2024-03-29T10:40:08.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.850+0000 7ff697251640 1 -- 172.21.15.114:0/38518825 >> v1:172.21.15.114:6816/775044307 conn(0x7ff67c06de90 legacy=0x7ff67c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:08.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.850+0000 7ff697251640 10 monclient: shutdown 2024-03-29T10:40:08.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.850+0000 7ff697251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:08.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.850+0000 7ff697251640 1 -- 172.21.15.114:0/38518825 >> v1:172.21.15.176:6789/0 conn(0x7ff6901457f0 legacy=0x7ff6901eec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:08.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.853+0000 7ff697251640 1 -- 172.21.15.114:0/38518825 shutdown_connections 2024-03-29T10:40:08.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.853+0000 7ff697251640 1 -- 172.21.15.114:0/38518825 >> 172.21.15.114:0/38518825 conn(0x7ff690066a00 msgr2=0x7ff69013e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:08.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.853+0000 7ff697251640 1 -- 172.21.15.114:0/38518825 shutdown_connections 2024-03-29T10:40:08.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.853+0000 7ff697251640 1 -- 172.21.15.114:0/38518825 wait complete. 2024-03-29T10:40:08.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:08.853+0000 7ff697251640 1 librados: shutdown 2024-03-29T10:40:08.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:15.662 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:15.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.824+0000 7f1bd34d6640 10 monclient: get_monmap_and_config 2024-03-29T10:40:15.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.824+0000 7f1bd34d6640 10 monclient: build_initial_monmap 2024-03-29T10:40:15.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.824+0000 7f1bd34d6640 10 monclient: monmap: 2024-03-29T10:40:15.810 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:15.810 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:15.810 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:40:15.826192+0000 2024-03-29T10:40:15.810 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:40:15.826192+0000 2024-03-29T10:40:15.810 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:15.810 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:15.810 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:15.810 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:40:15.811 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:40:15.811 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:15.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.826+0000 7f1bd34d6640 1 Processor -- start 2024-03-29T10:40:15.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.826+0000 7f1bd34d6640 1 -- start start 2024-03-29T10:40:15.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.826+0000 7f1bd34d6640 10 monclient: init 2024-03-29T10:40:15.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.827+0000 7f1bd34d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:15.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.827+0000 7f1bd34d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:15.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.827+0000 7f1bd34d6640 10 monclient(hunting): picked mon.noname-c con 0x7f1bcc0a9930 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:15.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.827+0000 7f1bd34d6640 10 monclient(hunting): picked mon.noname-b con 0x7f1bcc149dc0 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:15.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.827+0000 7f1bd34d6640 10 monclient(hunting): picked mon.noname-a con 0x7f1bcc14ff40 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:15.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.827+0000 7f1bd34d6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc0aae60 con 0x7f1bcc14ff40 2024-03-29T10:40:15.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.827+0000 7f1bd34d6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc0b4310 con 0x7f1bcc0a9930 2024-03-29T10:40:15.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.827+0000 7f1bd34d6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc0afdd0 con 0x7f1bcc149dc0 2024-03-29T10:40:15.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.827+0000 7f1bd34d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:15.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.827+0000 7f1bd34d6640 10 monclient(hunting): authenticate will time out at 881.068750s 2024-03-29T10:40:15.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.834+0000 7f1bd0a71640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f1bcc149dc0 0x7f1bcc14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33595/0 (socket says 172.21.15.114:33595) 2024-03-29T10:40:15.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.834+0000 7f1bd0a71640 1 -- 172.21.15.114:0/1196852474 learned_addr learned my addr 172.21.15.114:0/1196852474 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:15.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.834+0000 7f1bc3fff640 1 -- 172.21.15.114:0/1196852474 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337373053 0 0) 0x7f1bcc0afdd0 con 0x7f1bcc149dc0 2024-03-29T10:40:15.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.834+0000 7f1bc3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:15.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.834+0000 7f1bc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:15.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.834+0000 7f1bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.834+0000 7f1bc3fff640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.834+0000 7f1bc3fff640 1 -- 172.21.15.114:0/1196852474 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb8003520 con 0x7f1bcc149dc0 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.834+0000 7f1bc3fff640 1 -- 172.21.15.114:0/1196852474 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3474044540 0 0) 0x7f1bb8003520 con 0x7f1bcc149dc0 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.834+0000 7f1bc3fff640 1 -- 172.21.15.114:0/1196852474 >> v1:172.21.15.176:6789/0 conn(0x7f1bcc0a9930 legacy=0x7f1bcc143a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 1 -- 172.21.15.114:0/1196852474 >> v1:172.21.15.114:6789/0 conn(0x7f1bcc14ff40 legacy=0x7f1bcc152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 1 monclient: found mon.noname-b 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 1 -- 172.21.15.114:0/1196852474 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc14a950 con 0x7f1bcc149dc0 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 10 monclient: _finish_auth 0 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:15.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bd34d6640 5 monclient: authenticate success, global_id 4259 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bd34d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 1 -- 172.21.15.114:0/1196852474 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1bc4001a10 con 0x7f1bcc149dc0 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 10 monclient: dump: 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:15.822 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bd34d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 1 -- 172.21.15.114:0/1196852474 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bc4001ce0 con 0x7f1bcc149dc0 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 1 -- 172.21.15.114:0/1196852474 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1bc4002070 con 0x7f1bcc149dc0 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bc3fff640 10 monclient: dump: 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:15.823 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:15.824 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:15.824 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:15.824 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:15.824 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:15.824 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:15.824 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:15.824 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:15.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bd34d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:15.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.835+0000 7f1bd34d6640 4 set_mon_vals no callback set 2024-03-29T10:40:15.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.836+0000 7f1bd34d6640 10 monclient: shutdown 2024-03-29T10:40:15.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.836+0000 7f1bd34d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:15.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.836+0000 7f1bd34d6640 1 -- 172.21.15.114:0/1196852474 >> v1:172.21.15.114:6790/0 conn(0x7f1bcc149dc0 legacy=0x7f1bcc14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:15.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.839+0000 7f1bd34d6640 1 -- 172.21.15.114:0/1196852474 shutdown_connections 2024-03-29T10:40:15.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.839+0000 7f1bd34d6640 1 -- 172.21.15.114:0/1196852474 >> 172.21.15.114:0/1196852474 conn(0x7f1bcc0b35e0 msgr2=0x7f1bcc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:15.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.839+0000 7f1bd34d6640 1 -- 172.21.15.114:0/1196852474 shutdown_connections 2024-03-29T10:40:15.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.839+0000 7f1bd34d6640 1 -- 172.21.15.114:0/1196852474 wait complete. 2024-03-29T10:40:15.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.840+0000 7f1bd34d6640 10 monclient: build_initial_monmap 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.840+0000 7f1bd34d6640 10 monclient: monmap: 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.840+0000 7f1bd34d6640 1 librados: starting msgr at 2024-03-29T10:40:15.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.840+0000 7f1bd34d6640 1 librados: starting objecter 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.840+0000 7f1bd34d6640 1 Processor -- start 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.840+0000 7f1bd34d6640 1 -- start start 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.840+0000 7f1bd34d6640 1 librados: setting wanted keys 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.840+0000 7f1bd34d6640 1 librados: calling monclient init 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.840+0000 7f1bd34d6640 10 monclient: init 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.841+0000 7f1bd34d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.841+0000 7f1bd34d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.841+0000 7f1bd34d6640 10 monclient(hunting): picked mon.noname-a con 0x7f1bcc0a9930 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.841+0000 7f1bd34d6640 10 monclient(hunting): picked mon.noname-c con 0x7f1bcc149dc0 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.841+0000 7f1bd34d6640 10 monclient(hunting): picked mon.noname-b con 0x7f1bcc14ff40 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.841+0000 7f1bd34d6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc140590 con 0x7f1bcc0a9930 2024-03-29T10:40:15.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.841+0000 7f1bd34d6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc13f3b0 con 0x7f1bcc14ff40 2024-03-29T10:40:15.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.841+0000 7f1bd34d6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc13f540 con 0x7f1bcc149dc0 2024-03-29T10:40:15.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.841+0000 7f1bd34d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:15.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.841+0000 7f1bd34d6640 10 monclient(hunting): authenticate will time out at 881.082781s 2024-03-29T10:40:15.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.847+0000 7f1bd1272640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1bcc0a9930 0x7f1bcc13fcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52225/0 (socket says 172.21.15.114:52225) 2024-03-29T10:40:15.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.847+0000 7f1bd1272640 1 -- 172.21.15.114:0/989719068 learned_addr learned my addr 172.21.15.114:0/989719068 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:15.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.848+0000 7f1bc1ffb640 10 client.?.objecter ms_handle_connect 0x7f1bcc0a9930 2024-03-29T10:40:15.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.848+0000 7f1bc1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:15.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.848+0000 7f1bc1ffb640 10 client.?.objecter ms_handle_connect 0x7f1bcc149dc0 2024-03-29T10:40:15.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.848+0000 7f1bc1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:15.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.848+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654765311 0 0) 0x7f1bcc140590 con 0x7f1bcc0a9930 2024-03-29T10:40:15.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.848+0000 7f1bc1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:15.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.848+0000 7f1bc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:15.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.848+0000 7f1bc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:15.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.848+0000 7f1bc1ffb640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:40:15.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.848+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb0004eb0 con 0x7f1bcc0a9930 2024-03-29T10:40:15.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730165136 0 0) 0x7f1bcc13f540 con 0x7f1bcc149dc0 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb00051d0 con 0x7f1bcc149dc0 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1204884752 0 0) 0x7f1bb0004eb0 con 0x7f1bcc0a9930 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bb0005640 con 0x7f1bcc0a9930 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1bbc001830 con 0x7f1bcc0a9930 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 10 monclient(hunting): dump: 2024-03-29T10:40:15.834 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2406924656 0 0) 0x7f1bb00051d0 con 0x7f1bcc149dc0 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bb0005d30 con 0x7f1bcc149dc0 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1bc4001830 con 0x7f1bcc149dc0 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:15.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.849+0000 7f1bc1ffb640 10 monclient(hunting): dump: 2024-03-29T10:40:15.836 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:15.836 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:15.836 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:15.836 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:15.836 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:15.836 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:15.836 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:15.836 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:15.837 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:15.837 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:15.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4108032545 0 0) 0x7f1bb0005d30 con 0x7f1bcc149dc0 2024-03-29T10:40:15.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 >> v1:172.21.15.176:6789/0 conn(0x7f1bcc14ff40 legacy=0x7f1bcc1e61f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:15.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 >> v1:172.21.15.114:6789/0 conn(0x7f1bcc0a9930 legacy=0x7f1bcc13fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:15.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bc1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:15.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bc1ffb640 1 monclient: found mon.c 2024-03-29T10:40:15.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bc1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc148890 con 0x7f1bcc149dc0 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bc1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bc1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bd34d6640 5 monclient: authenticate success, global_id 4262 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bd34d6640 10 monclient: _renew_subs 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bd34d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.850+0000 7f1bd34d6640 1 -- 172.21.15.114:0/989719068 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bcc148a80 con 0x7f1bcc149dc0 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.851+0000 7f1bd34d6640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.851+0000 7f1bd34d6640 10 monclient: _renew_subs 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.851+0000 7f1bd34d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.851+0000 7f1bd34d6640 1 -- 172.21.15.114:0/989719068 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bcc1df530 con 0x7f1bcc149dc0 2024-03-29T10:40:15.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.851+0000 7f1bd34d6640 1 librados: init done 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.852+0000 7f1bd34d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.853+0000 7f1bd34d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.853+0000 7f1bd34d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.853+0000 7f1bd34d6640 1 -- 172.21.15.114:0/989719068 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bcc140740 con 0x7f1bcc149dc0 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.853+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bc4004d00 con 0x7f1bcc149dc0 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.853+0000 7f1bc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.853+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1bc4001830 con 0x7f1bcc149dc0 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.853+0000 7f1bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.853+0000 7f1bc1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.853+0000 7f1bc1ffb640 10 monclient: dump: 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:15.841 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:15.842 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:15.842 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:15.842 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:15.842 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:15.842 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:15.842 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:15.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.853+0000 7f1bc37fe640 4 set_mon_vals no callback set 2024-03-29T10:40:15.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.855+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f1bc401ace0 con 0x7f1bcc149dc0 2024-03-29T10:40:15.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.856+0000 7f1bd1272640 10 monclient: get_auth_request con 0x7f1bb006e850 auth_method 0 2024-03-29T10:40:15.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.856+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f1bc40546a0 con 0x7f1bcc149dc0 2024-03-29T10:40:15.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.856+0000 7f1bc1ffb640 10 client.4262.objecter ms_dispatch 0x7f1bcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:15.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.856+0000 7f1bc1ffb640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:15.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.856+0000 7f1bc1ffb640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:15.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.856+0000 7f1bc1ffb640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T10:40:15.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.856+0000 7f1bc1ffb640 10 client.4262.objecter ms_handle_connect 0x7f1bb006e850 2024-03-29T10:40:15.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.863+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bc4050940 con 0x7f1bcc149dc0 2024-03-29T10:40:15.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.863+0000 7f1bc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:15.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:15.863+0000 7f1bc1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:16.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.201+0000 7f1bd34d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:16.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.201+0000 7f1bd34d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:16.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.201+0000 7f1bd34d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:16.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.201+0000 7f1bd34d6640 1 -- 172.21.15.114:0/989719068 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bcc148c10 con 0x7f1bcc149dc0 2024-03-29T10:40:16.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.202+0000 7f1bc1ffb640 1 -- 172.21.15.114:0/989719068 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f1bc4050ae0 con 0x7f1bcc149dc0 2024-03-29T10:40:16.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.202+0000 7f1bc1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:16.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.202+0000 7f1bc1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:16.187 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:40:16.187 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:16.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.205+0000 7f1bd34d6640 10 librados: watch_flush enter 2024-03-29T10:40:16.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.206+0000 7f1bd34d6640 10 librados: watch_flush exit 2024-03-29T10:40:16.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.206+0000 7f1bd34d6640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T10:40:16.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.206+0000 7f1bd34d6640 10 client.4262.objecter successfully canceled tick 2024-03-29T10:40:16.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.206+0000 7f1bd34d6640 1 -- 172.21.15.114:0/989719068 >> v1:172.21.15.114:6816/775044307 conn(0x7f1bb006e850 legacy=0x7f1bb0070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:16.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.206+0000 7f1bd34d6640 10 monclient: shutdown 2024-03-29T10:40:16.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.206+0000 7f1bd34d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:16.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.206+0000 7f1bd34d6640 1 -- 172.21.15.114:0/989719068 >> v1:172.21.15.114:6790/0 conn(0x7f1bcc149dc0 legacy=0x7f1bcc1dec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:16.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.209+0000 7f1bd34d6640 1 -- 172.21.15.114:0/989719068 shutdown_connections 2024-03-29T10:40:16.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.209+0000 7f1bd34d6640 1 -- 172.21.15.114:0/989719068 >> 172.21.15.114:0/989719068 conn(0x7f1bcc0b35e0 msgr2=0x7f1bcc0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:16.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.209+0000 7f1bd34d6640 1 -- 172.21.15.114:0/989719068 shutdown_connections 2024-03-29T10:40:16.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.209+0000 7f1bd34d6640 1 -- 172.21.15.114:0/989719068 wait complete. 2024-03-29T10:40:16.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:16.210+0000 7f1bd34d6640 1 librados: shutdown 2024-03-29T10:40:16.211 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:23.018 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:23.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.176+0000 7fa8763fb640 10 monclient: get_monmap_and_config 2024-03-29T10:40:23.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.176+0000 7fa8763fb640 10 monclient: build_initial_monmap 2024-03-29T10:40:23.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.176+0000 7fa8763fb640 10 monclient: monmap: 2024-03-29T10:40:23.160 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:23.160 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:23.160 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:40:23.177506+0000 2024-03-29T10:40:23.160 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:40:23.177506+0000 2024-03-29T10:40:23.160 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:23.160 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:23.161 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:23.161 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:40:23.161 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:40:23.161 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:23.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.177+0000 7fa8763fb640 1 Processor -- start 2024-03-29T10:40:23.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.177+0000 7fa8763fb640 1 -- start start 2024-03-29T10:40:23.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.177+0000 7fa8763fb640 10 monclient: init 2024-03-29T10:40:23.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.178+0000 7fa8763fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:23.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.178+0000 7fa8763fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:23.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.178+0000 7fa8763fb640 10 monclient(hunting): picked mon.noname-a con 0x7fa8700b1f60 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:23.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.178+0000 7fa8763fb640 10 monclient(hunting): picked mon.noname-b con 0x7fa870071eb0 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:23.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.178+0000 7fa8763fb640 10 monclient(hunting): picked mon.noname-c con 0x7fa870143d10 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:23.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.178+0000 7fa8763fb640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87006ef80 con 0x7fa8700b1f60 2024-03-29T10:40:23.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.178+0000 7fa8763fb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87006f260 con 0x7fa870143d10 2024-03-29T10:40:23.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.178+0000 7fa8763fb640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8700b40c0 con 0x7fa870071eb0 2024-03-29T10:40:23.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.178+0000 7fa8763fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:23.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.178+0000 7fa8763fb640 10 monclient(hunting): authenticate will time out at 888.419505s 2024-03-29T10:40:23.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.179+0000 7fa86f7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fa870071eb0 0x7fa870146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:59910/0 (socket says 172.21.15.114:59910) 2024-03-29T10:40:23.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.179+0000 7fa86f7fe640 1 -- 172.21.15.114:0/458153801 learned_addr learned my addr 172.21.15.114:0/458153801 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:23.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.179+0000 7fa86effd640 1 -- 172.21.15.114:0/458153801 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083558729 0 0) 0x7fa8700b40c0 con 0x7fa870071eb0 2024-03-29T10:40:23.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.180+0000 7fa86effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:23.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.180+0000 7fa86effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:23.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.180+0000 7fa86effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:23.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.180+0000 7fa86effd640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:40:23.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.180+0000 7fa86effd640 1 -- 172.21.15.114:0/458153801 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa85c003520 con 0x7fa870071eb0 2024-03-29T10:40:23.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.180+0000 7fa86effd640 1 -- 172.21.15.114:0/458153801 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3462086154 0 0) 0x7fa85c003520 con 0x7fa870071eb0 2024-03-29T10:40:23.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.180+0000 7fa86effd640 1 -- 172.21.15.114:0/458153801 >> v1:172.21.15.176:6789/0 conn(0x7fa870143d10 legacy=0x7fa870146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:23.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 1 -- 172.21.15.114:0/458153801 >> v1:172.21.15.114:6789/0 conn(0x7fa8700b1f60 legacy=0x7fa870143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:23.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:23.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 1 monclient: found mon.noname-b 2024-03-29T10:40:23.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 1 -- 172.21.15.114:0/458153801 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa870155d70 con 0x7fa870071eb0 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 10 monclient: _finish_auth 0 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa8763fb640 5 monclient: authenticate success, global_id 4265 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa8763fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 1 -- 172.21.15.114:0/458153801 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa860001690 con 0x7fa870071eb0 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa86effd640 10 monclient: dump: 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:23.166 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:23.167 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:23.167 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:23.167 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:23.167 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:23.167 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:23.167 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:23.167 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:23.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.181+0000 7fa8763fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:23.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.182+0000 7fa86effd640 1 -- 172.21.15.114:0/458153801 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa860001920 con 0x7fa870071eb0 2024-03-29T10:40:23.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.182+0000 7fa86effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:23.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.182+0000 7fa8763fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:23.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.182+0000 7fa8763fb640 4 set_mon_vals no callback set 2024-03-29T10:40:23.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.182+0000 7fa86effd640 1 -- 172.21.15.114:0/458153801 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa860001cb0 con 0x7fa870071eb0 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.182+0000 7fa86effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.182+0000 7fa86effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.182+0000 7fa86effd640 10 monclient: dump: 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:23.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.183+0000 7fa8763fb640 10 monclient: shutdown 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.183+0000 7fa8763fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.183+0000 7fa8763fb640 1 -- 172.21.15.114:0/458153801 >> v1:172.21.15.114:6790/0 conn(0x7fa870071eb0 legacy=0x7fa870146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.183+0000 7fa8763fb640 1 -- 172.21.15.114:0/458153801 shutdown_connections 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.183+0000 7fa8763fb640 1 -- 172.21.15.114:0/458153801 >> 172.21.15.114:0/458153801 conn(0x7fa870066a00 msgr2=0x7fa870066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.184+0000 7fa8763fb640 1 -- 172.21.15.114:0/458153801 shutdown_connections 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.184+0000 7fa8763fb640 1 -- 172.21.15.114:0/458153801 wait complete. 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.185+0000 7fa8763fb640 10 monclient: build_initial_monmap 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.185+0000 7fa8763fb640 10 monclient: monmap: 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:23.171 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:23.172 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:23.172 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:40:23.172 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:40:23.172 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:23.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.185+0000 7fa8763fb640 1 librados: starting msgr at 2024-03-29T10:40:23.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.185+0000 7fa8763fb640 1 librados: starting objecter 2024-03-29T10:40:23.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.185+0000 7fa8763fb640 1 Processor -- start 2024-03-29T10:40:23.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.186+0000 7fa8763fb640 1 -- start start 2024-03-29T10:40:23.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.186+0000 7fa8763fb640 1 librados: setting wanted keys 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.186+0000 7fa8763fb640 1 librados: calling monclient init 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.186+0000 7fa8763fb640 10 monclient: init 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.187+0000 7fa8763fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.187+0000 7fa8763fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.187+0000 7fa8763fb640 10 monclient(hunting): picked mon.noname-c con 0x7fa8700b1f60 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.187+0000 7fa8763fb640 10 monclient(hunting): picked mon.noname-a con 0x7fa870143d10 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.187+0000 7fa8763fb640 10 monclient(hunting): picked mon.noname-b con 0x7fa870070920 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.187+0000 7fa8763fb640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8701551e0 con 0x7fa870143d10 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.187+0000 7fa8763fb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8701422b0 con 0x7fa870070920 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.187+0000 7fa8763fb640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa870154c30 con 0x7fa8700b1f60 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.187+0000 7fa8763fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.187+0000 7fa8763fb640 10 monclient(hunting): authenticate will time out at 888.428722s 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.188+0000 7fa86ffff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fa8700b1f60 0x7fa870141ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33072/0 (socket says 172.21.15.114:33072) 2024-03-29T10:40:23.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.188+0000 7fa86ffff640 1 -- 172.21.15.114:0/4001687432 learned_addr learned my addr 172.21.15.114:0/4001687432 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:23.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.188+0000 7fa86cff9640 10 client.?.objecter ms_handle_connect 0x7fa8700b1f60 2024-03-29T10:40:23.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.188+0000 7fa86cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:23.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.190+0000 7fa86cff9640 10 client.?.objecter ms_handle_connect 0x7fa870143d10 2024-03-29T10:40:23.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.190+0000 7fa86cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:23.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.190+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646672065 0 0) 0x7fa8701551e0 con 0x7fa870143d10 2024-03-29T10:40:23.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.190+0000 7fa86cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:23.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.190+0000 7fa86cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:23.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.190+0000 7fa86cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:23.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.190+0000 7fa86cff9640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:40:23.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.190+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa850004eb0 con 0x7fa870143d10 2024-03-29T10:40:23.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.191+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823169543 0 0) 0x7fa870154c30 con 0x7fa8700b1f60 2024-03-29T10:40:23.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.191+0000 7fa86cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:23.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.191+0000 7fa86cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:23.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.191+0000 7fa86cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:23.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.191+0000 7fa86cff9640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:40:23.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.191+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8701551e0 con 0x7fa8700b1f60 2024-03-29T10:40:23.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 10 client.?.objecter ms_handle_connect 0x7fa870070920 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389208937 0 0) 0x7fa8701422b0 con 0x7fa870070920 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa870154c30 con 0x7fa870070920 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2007220294 0 0) 0x7fa870154c30 con 0x7fa870070920 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8701422b0 con 0x7fa870070920 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa858001650 con 0x7fa870070920 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:23.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 10 monclient(hunting): dump: 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.192+0000 7fa86cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1276337135 0 0) 0x7fa850004eb0 con 0x7fa870143d10 2024-03-29T10:40:23.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa870154c30 con 0x7fa870143d10 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4135390994 0 0) 0x7fa8701422b0 con 0x7fa870070920 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 >> v1:172.21.15.114:6790/0 conn(0x7fa8700b1f60 legacy=0x7fa870141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 >> v1:172.21.15.114:6789/0 conn(0x7fa870143d10 legacy=0x7fa870133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 1 monclient: found mon.b 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa870133e40 con 0x7fa870070920 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 10 monclient: _finish_auth 0 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa858001b60 con 0x7fa870070920 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:23.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa8580035f0 con 0x7fa870070920 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86cff9640 10 monclient: dump: 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:23.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.193+0000 7fa86e7fc640 4 set_mon_vals no callback set 2024-03-29T10:40:23.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.194+0000 7fa8763fb640 5 monclient: authenticate success, global_id 4261 2024-03-29T10:40:23.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.194+0000 7fa8763fb640 10 monclient: _renew_subs 2024-03-29T10:40:23.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.194+0000 7fa8763fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:23.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.194+0000 7fa8763fb640 1 -- 172.21.15.114:0/4001687432 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa870071280 con 0x7fa870070920 2024-03-29T10:40:23.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.194+0000 7fa8763fb640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:23.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.194+0000 7fa8763fb640 10 monclient: _renew_subs 2024-03-29T10:40:23.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.194+0000 7fa8763fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:23.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.194+0000 7fa8763fb640 1 -- 172.21.15.114:0/4001687432 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa870153980 con 0x7fa870070920 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.194+0000 7fa8763fb640 1 librados: init done 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.196+0000 7fa8763fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.196+0000 7fa8763fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.196+0000 7fa8763fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.196+0000 7fa8763fb640 1 -- 172.21.15.114:0/4001687432 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa870071620 con 0x7fa870070920 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.197+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fa8580195c0 con 0x7fa870070920 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.198+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fa858083520 con 0x7fa870070920 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.198+0000 7fa86cff9640 10 client.4261.objecter ms_dispatch 0x7fa870000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.198+0000 7fa86cff9640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.198+0000 7fa86cff9640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.198+0000 7fa86cff9640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.198+0000 7fa86ffff640 10 monclient: get_auth_request con 0x7fa85006dec0 auth_method 0 2024-03-29T10:40:23.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.198+0000 7fa86cff9640 10 client.4261.objecter ms_handle_connect 0x7fa85006dec0 2024-03-29T10:40:23.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.206+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa85804ee00 con 0x7fa870070920 2024-03-29T10:40:23.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.206+0000 7fa86cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:23.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.206+0000 7fa86cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:23.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.547+0000 7fa8763fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:23.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.547+0000 7fa8763fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:23.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.547+0000 7fa8763fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:23.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.547+0000 7fa8763fb640 1 -- 172.21.15.114:0/4001687432 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa870071410 con 0x7fa870070920 2024-03-29T10:40:23.533 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.548+0000 7fa86cff9640 1 -- 172.21.15.114:0/4001687432 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fa858050d30 con 0x7fa870070920 2024-03-29T10:40:23.533 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.548+0000 7fa86cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:23.533 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.548+0000 7fa86cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:23.536 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:40:23.536 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:23.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.556+0000 7fa8763fb640 10 librados: watch_flush enter 2024-03-29T10:40:23.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.556+0000 7fa8763fb640 10 librados: watch_flush exit 2024-03-29T10:40:23.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.556+0000 7fa8763fb640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T10:40:23.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.556+0000 7fa8763fb640 10 client.4261.objecter successfully canceled tick 2024-03-29T10:40:23.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.556+0000 7fa8763fb640 1 -- 172.21.15.114:0/4001687432 >> v1:172.21.15.114:6816/775044307 conn(0x7fa85006dec0 legacy=0x7fa850070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:23.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.556+0000 7fa8763fb640 10 monclient: shutdown 2024-03-29T10:40:23.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.556+0000 7fa8763fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:23.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.556+0000 7fa8763fb640 1 -- 172.21.15.114:0/4001687432 >> v1:172.21.15.176:6789/0 conn(0x7fa870070920 legacy=0x7fa870133730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:23.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.556+0000 7fa8763fb640 1 -- 172.21.15.114:0/4001687432 shutdown_connections 2024-03-29T10:40:23.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.556+0000 7fa8763fb640 1 -- 172.21.15.114:0/4001687432 >> 172.21.15.114:0/4001687432 conn(0x7fa870066a00 msgr2=0x7fa8700b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:23.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.557+0000 7fa8763fb640 1 -- 172.21.15.114:0/4001687432 shutdown_connections 2024-03-29T10:40:23.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.557+0000 7fa8763fb640 1 -- 172.21.15.114:0/4001687432 wait complete. 2024-03-29T10:40:23.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:23.557+0000 7fa8763fb640 1 librados: shutdown 2024-03-29T10:40:23.557 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:30.363 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:30.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.542+0000 7fc4c093e640 10 monclient: get_monmap_and_config 2024-03-29T10:40:30.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.542+0000 7fc4c093e640 10 monclient: build_initial_monmap 2024-03-29T10:40:30.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.543+0000 7fc4c093e640 10 monclient: monmap: 2024-03-29T10:40:30.527 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:30.527 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:30.527 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:40:30.543277+0000 2024-03-29T10:40:30.527 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:40:30.543277+0000 2024-03-29T10:40:30.527 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:30.527 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:30.528 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:30.528 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:40:30.528 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:40:30.528 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:30.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.544+0000 7fc4c093e640 1 Processor -- start 2024-03-29T10:40:30.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.544+0000 7fc4c093e640 1 -- start start 2024-03-29T10:40:30.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.544+0000 7fc4c093e640 10 monclient: init 2024-03-29T10:40:30.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4c093e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:30.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4c093e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:30.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4c093e640 10 monclient(hunting): picked mon.noname-c con 0x7fc4bc0b0770 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:30.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4c093e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4bc14a700 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:30.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4c093e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4bc0ac5a0 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:30.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4c093e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc0aba70 con 0x7fc4bc14a700 2024-03-29T10:40:30.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4c093e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc0aecb0 con 0x7fc4bc0b0770 2024-03-29T10:40:30.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4c093e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc0af150 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4c093e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:30.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4c093e640 10 monclient(hunting): authenticate will time out at 895.786494s 2024-03-29T10:40:30.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4ba59c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fc4bc0b0770 0x7fc4bc149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:52192/0 (socket says 172.21.15.114:52192) 2024-03-29T10:40:30.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.546+0000 7fc4ba59c640 1 -- 172.21.15.114:0/1789132350 learned_addr learned my addr 172.21.15.114:0/1789132350 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:30.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 1 -- 172.21.15.114:0/1789132350 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058189531 0 0) 0x7fc4bc0aecb0 con 0x7fc4bc0b0770 2024-03-29T10:40:30.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:30.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:30.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:30.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient(hunting): my global_id is 4264 2024-03-29T10:40:30.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 1 -- 172.21.15.114:0/1789132350 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a4003520 con 0x7fc4bc0b0770 2024-03-29T10:40:30.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 1 -- 172.21.15.114:0/1789132350 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265276431 0 0) 0x7fc4bc0aba70 con 0x7fc4bc14a700 2024-03-29T10:40:30.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:30.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:30.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:30.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:40:30.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 1 -- 172.21.15.114:0/1789132350 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a4003840 con 0x7fc4bc14a700 2024-03-29T10:40:30.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 1 -- 172.21.15.114:0/1789132350 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2971148298 0 0) 0x7fc4a4003520 con 0x7fc4bc0b0770 2024-03-29T10:40:30.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 1 -- 172.21.15.114:0/1789132350 >> v1:172.21.15.114:6790/0 conn(0x7fc4bc0ac5a0 legacy=0x7fc4bc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:30.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 1 -- 172.21.15.114:0/1789132350 >> v1:172.21.15.114:6789/0 conn(0x7fc4bc14a700 legacy=0x7fc4bc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:30.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient: _finish_hunting 0 2024-03-29T10:40:30.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 1 monclient: found mon.noname-c 2024-03-29T10:40:30.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:30.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:40:30.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 1 -- 172.21.15.114:0/1789132350 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc0ac9d0 con 0x7fc4bc0b0770 2024-03-29T10:40:30.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient: _finish_auth 0 2024-03-29T10:40:30.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 10 monclient: _check_auth_tickets 2024-03-29T10:40:30.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:30.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4c093e640 5 monclient: authenticate success, global_id 4264 2024-03-29T10:40:30.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.547+0000 7fc4c093e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:30.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4b959a640 1 -- 172.21.15.114:0/1789132350 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc4b0001670 con 0x7fc4bc0b0770 2024-03-29T10:40:30.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:30.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4b959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:30.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4b959a640 10 monclient: dump: 2024-03-29T10:40:30.548 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:30.548 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:30.548 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:30.549 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:30.549 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:30.549 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:30.549 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:30.549 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:30.549 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:30.549 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:30.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4c093e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:30.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4b959a640 1 -- 172.21.15.114:0/1789132350 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b00018c0 con 0x7fc4bc0b0770 2024-03-29T10:40:30.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:30.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4b959a640 1 -- 172.21.15.114:0/1789132350 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc4b0001c50 con 0x7fc4bc0b0770 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4b959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4b959a640 10 monclient: dump: 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:30.550 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4c093e640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.548+0000 7fc4c093e640 4 set_mon_vals no callback set 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.549+0000 7fc4c093e640 10 monclient: shutdown 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.549+0000 7fc4c093e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.549+0000 7fc4c093e640 1 -- 172.21.15.114:0/1789132350 >> v1:172.21.15.176:6789/0 conn(0x7fc4bc0b0770 legacy=0x7fc4bc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.549+0000 7fc4c093e640 1 -- 172.21.15.114:0/1789132350 shutdown_connections 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.549+0000 7fc4c093e640 1 -- 172.21.15.114:0/1789132350 >> 172.21.15.114:0/1789132350 conn(0x7fc4bc06b660 msgr2=0x7fc4bc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.549+0000 7fc4c093e640 1 -- 172.21.15.114:0/1789132350 shutdown_connections 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.549+0000 7fc4c093e640 1 -- 172.21.15.114:0/1789132350 wait complete. 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.552+0000 7fc4c093e640 10 monclient: build_initial_monmap 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.552+0000 7fc4c093e640 10 monclient: monmap: 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:30.551 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:30.552 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:40:30.552 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:40:30.552 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:30.552 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:30.552 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:30.552 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:40:30.552 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:40:30.552 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:30.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.553+0000 7fc4c093e640 1 librados: starting msgr at 2024-03-29T10:40:30.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.553+0000 7fc4c093e640 1 librados: starting objecter 2024-03-29T10:40:30.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.553+0000 7fc4c093e640 1 Processor -- start 2024-03-29T10:40:30.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.553+0000 7fc4c093e640 1 -- start start 2024-03-29T10:40:30.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.553+0000 7fc4c093e640 1 librados: setting wanted keys 2024-03-29T10:40:30.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.553+0000 7fc4c093e640 1 librados: calling monclient init 2024-03-29T10:40:30.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.553+0000 7fc4c093e640 10 monclient: init 2024-03-29T10:40:30.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4c093e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:30.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4c093e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:30.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4c093e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4bc0ac5a0 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:30.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4c093e640 10 monclient(hunting): picked mon.noname-c con 0x7fc4bc0b0770 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:30.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4c093e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4bc14a700 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:30.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4c093e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc1458c0 con 0x7fc4bc14a700 2024-03-29T10:40:30.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4c093e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc1283e0 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4c093e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc1263a0 con 0x7fc4bc0b0770 2024-03-29T10:40:30.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4c093e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:30.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4c093e640 10 monclient(hunting): authenticate will time out at 895.795337s 2024-03-29T10:40:30.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4ba59c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fc4bc0ac5a0 0x7fc4bc1268c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:52476/0 (socket says 172.21.15.114:52476) 2024-03-29T10:40:30.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4ba59c640 1 -- 172.21.15.114:0/3230869801 learned_addr learned my addr 172.21.15.114:0/3230869801 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:30.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4aaffd640 10 client.?.objecter ms_handle_connect 0x7fc4bc0ac5a0 2024-03-29T10:40:30.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:30.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4aaffd640 10 client.?.objecter ms_handle_connect 0x7fc4bc0b0770 2024-03-29T10:40:30.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.555+0000 7fc4aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 client.?.objecter ms_handle_connect 0x7fc4bc14a700 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761928418 0 0) 0x7fc4bc1283e0 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc49c004eb0 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096823622 0 0) 0x7fc4bc1263a0 con 0x7fc4bc0b0770 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:30.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4bc1283e0 con 0x7fc4bc0b0770 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3806296875 0 0) 0x7fc49c004eb0 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4bc1263a0 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983093784 0 0) 0x7fc4bc1458c0 con 0x7fc4bc14a700 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc49c004eb0 con 0x7fc4bc14a700 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc4b0001630 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): dump: 2024-03-29T10:40:30.556 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.556+0000 7fc4aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1942488742 0 0) 0x7fc4bc1263a0 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 >> v1:172.21.15.114:6790/0 conn(0x7fc4bc0b0770 legacy=0x7fc4bc144aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:30.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 >> v1:172.21.15.114:6789/0 conn(0x7fc4bc14a700 legacy=0x7fc4bc1451b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 1 monclient: found mon.b 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc1239d0 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 10 monclient: _finish_auth 0 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4c093e640 5 monclient: authenticate success, global_id 4267 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4c093e640 10 monclient: _renew_subs 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4c093e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4c093e640 1 -- 172.21.15.114:0/3230869801 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4bc1dc380 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4c093e640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4c093e640 10 monclient: _renew_subs 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4c093e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4c093e640 1 -- 172.21.15.114:0/3230869801 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc1271b0 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4c093e640 1 librados: init done 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b0004b00 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.557+0000 7fc4b8d99640 4 set_mon_vals no callback set 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.558+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc4b0004f10 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.558+0000 7fc4aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.558+0000 7fc4aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.558+0000 7fc4aaffd640 10 monclient: dump: 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:30.559 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.559+0000 7fc4c093e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.559+0000 7fc4c093e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.559+0000 7fc4c093e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.559+0000 7fc4c093e640 1 -- 172.21.15.114:0/3230869801 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a0005080 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.560+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fc4b0005070 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.560+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fc4b0084b50 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.560+0000 7fc4aaffd640 10 client.4267.objecter ms_dispatch 0x7fc4bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:30.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.560+0000 7fc4aaffd640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:30.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.560+0000 7fc4aaffd640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:30.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.560+0000 7fc4aaffd640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-29T10:40:30.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.560+0000 7fc4b9d9b640 10 monclient: get_auth_request con 0x7fc49c06ddf0 auth_method 0 2024-03-29T10:40:30.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.561+0000 7fc4aaffd640 10 client.4267.objecter ms_handle_connect 0x7fc49c06ddf0 2024-03-29T10:40:30.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.567+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4b0050430 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.568+0000 7fc4aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:30.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.568+0000 7fc4aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:30.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.925+0000 7fc4c093e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:30.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.925+0000 7fc4c093e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:30.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.925+0000 7fc4c093e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:30.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.925+0000 7fc4c093e640 1 -- 172.21.15.114:0/3230869801 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4a0005330 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.926+0000 7fc4aaffd640 1 -- 172.21.15.114:0/3230869801 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fc4b0052360 con 0x7fc4bc0ac5a0 2024-03-29T10:40:30.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.926+0000 7fc4aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:30.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.926+0000 7fc4aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:30.910 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:40:30.910 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:30.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.930+0000 7fc4c093e640 10 librados: watch_flush enter 2024-03-29T10:40:30.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.930+0000 7fc4c093e640 10 librados: watch_flush exit 2024-03-29T10:40:30.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.930+0000 7fc4c093e640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-29T10:40:30.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.930+0000 7fc4c093e640 10 client.4267.objecter successfully canceled tick 2024-03-29T10:40:30.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.930+0000 7fc4c093e640 1 -- 172.21.15.114:0/3230869801 >> v1:172.21.15.114:6816/775044307 conn(0x7fc49c06ddf0 legacy=0x7fc49c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:30.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.930+0000 7fc4c093e640 10 monclient: shutdown 2024-03-29T10:40:30.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.930+0000 7fc4c093e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:30.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.930+0000 7fc4c093e640 1 -- 172.21.15.114:0/3230869801 >> v1:172.21.15.176:6789/0 conn(0x7fc4bc0ac5a0 legacy=0x7fc4bc1268c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:30.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.931+0000 7fc4c093e640 1 -- 172.21.15.114:0/3230869801 shutdown_connections 2024-03-29T10:40:30.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.931+0000 7fc4c093e640 1 -- 172.21.15.114:0/3230869801 >> 172.21.15.114:0/3230869801 conn(0x7fc4bc06b660 msgr2=0x7fc4bc0abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:30.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.931+0000 7fc4c093e640 1 -- 172.21.15.114:0/3230869801 shutdown_connections 2024-03-29T10:40:30.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.931+0000 7fc4c093e640 1 -- 172.21.15.114:0/3230869801 wait complete. 2024-03-29T10:40:30.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:30.931+0000 7fc4c093e640 1 librados: shutdown 2024-03-29T10:40:30.929 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:37.736 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:37.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.888+0000 7fd2f0625640 10 monclient: get_monmap_and_config 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.888+0000 7fd2f0625640 10 monclient: build_initial_monmap 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.888+0000 7fd2f0625640 10 monclient: monmap: 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:40:37.890163+0000 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:40:37.890163+0000 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:40:37.874 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:37.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.889+0000 7fd2f0625640 1 Processor -- start 2024-03-29T10:40:37.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.889+0000 7fd2f0625640 1 -- start start 2024-03-29T10:40:37.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.889+0000 7fd2f0625640 10 monclient: init 2024-03-29T10:40:37.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.890+0000 7fd2f0625640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:37.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.890+0000 7fd2f0625640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:37.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.890+0000 7fd2f0625640 10 monclient(hunting): picked mon.noname-a con 0x7fd2e81457f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:37.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2f0625640 10 monclient(hunting): picked mon.noname-b con 0x7fd2e813f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:37.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ee3c1640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd2e81457f0 0x7fd2e813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:47356/0 (socket says 172.21.15.114:47356) 2024-03-29T10:40:37.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ee3c1640 1 -- 172.21.15.114:0/3744848389 learned_addr learned my addr 172.21.15.114:0/3744848389 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:37.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2f0625640 10 monclient(hunting): picked mon.noname-c con 0x7fd2e80a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:37.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2f0625640 1 -- 172.21.15.114:0/3744848389 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e80aaef0 con 0x7fd2e81457f0 2024-03-29T10:40:37.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2f0625640 1 -- 172.21.15.114:0/3744848389 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e8145bd0 con 0x7fd2e80a9d70 2024-03-29T10:40:37.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2f0625640 1 -- 172.21.15.114:0/3744848389 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e8145eb0 con 0x7fd2e813f650 2024-03-29T10:40:37.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2f0625640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2f0625640 10 monclient(hunting): authenticate will time out at 903.132659s 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 1 -- 172.21.15.114:0/3744848389 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371447300 0 0) 0x7fd2e80aaef0 con 0x7fd2e81457f0 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 10 monclient(hunting): _negotiate 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 1 -- 172.21.15.114:0/3744848389 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d4003520 con 0x7fd2e81457f0 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 1 -- 172.21.15.114:0/3744848389 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320490645 0 0) 0x7fd2e8145eb0 con 0x7fd2e813f650 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 10 monclient(hunting): _negotiate 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:40:37.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.891+0000 7fd2ed3bf640 1 -- 172.21.15.114:0/3744848389 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e80aaef0 con 0x7fd2e813f650 2024-03-29T10:40:37.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.892+0000 7fd2ed3bf640 1 -- 172.21.15.114:0/3744848389 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3023646726 0 0) 0x7fd2d4003520 con 0x7fd2e81457f0 2024-03-29T10:40:37.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.892+0000 7fd2ed3bf640 1 -- 172.21.15.114:0/3744848389 >> v1:172.21.15.114:6790/0 conn(0x7fd2e813f650 legacy=0x7fd2e80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:37.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.892+0000 7fd2ed3bf640 1 -- 172.21.15.114:0/3744848389 >> v1:172.21.15.176:6789/0 conn(0x7fd2e80a9d70 legacy=0x7fd2e814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:37.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 10 monclient: _finish_hunting 0 2024-03-29T10:40:37.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 1 monclient: found mon.noname-a 2024-03-29T10:40:37.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:37.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:40:37.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 1 -- 172.21.15.114:0/3744848389 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e8146340 con 0x7fd2e81457f0 2024-03-29T10:40:37.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 10 monclient: _finish_auth 0 2024-03-29T10:40:37.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 10 monclient: _check_auth_tickets 2024-03-29T10:40:37.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:37.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 1 -- 172.21.15.114:0/3744848389 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd2e4001690 con 0x7fd2e81457f0 2024-03-29T10:40:37.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 10 monclient: dump: 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:37.883 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2f0625640 5 monclient: authenticate success, global_id 4287 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2f0625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 1 -- 172.21.15.114:0/3744848389 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2e40017f0 con 0x7fd2e81457f0 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 1 -- 172.21.15.114:0/3744848389 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd2e4001c40 con 0x7fd2e81457f0 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2ed3bf640 10 monclient: dump: 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:37.884 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2f0625640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.894+0000 7fd2f0625640 4 set_mon_vals no callback set 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.895+0000 7fd2f0625640 10 monclient: shutdown 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.895+0000 7fd2f0625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.895+0000 7fd2f0625640 1 -- 172.21.15.114:0/3744848389 >> v1:172.21.15.114:6789/0 conn(0x7fd2e81457f0 legacy=0x7fd2e813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.895+0000 7fd2f0625640 1 -- 172.21.15.114:0/3744848389 shutdown_connections 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.895+0000 7fd2f0625640 1 -- 172.21.15.114:0/3744848389 >> 172.21.15.114:0/3744848389 conn(0x7fd2e8066a00 msgr2=0x7fd2e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.896+0000 7fd2f0625640 1 -- 172.21.15.114:0/3744848389 shutdown_connections 2024-03-29T10:40:37.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.896+0000 7fd2f0625640 1 -- 172.21.15.114:0/3744848389 wait complete. 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.897+0000 7fd2f0625640 10 monclient: build_initial_monmap 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.897+0000 7fd2f0625640 10 monclient: monmap: 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:37.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.897+0000 7fd2f0625640 1 librados: starting msgr at 2024-03-29T10:40:37.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.897+0000 7fd2f0625640 1 librados: starting objecter 2024-03-29T10:40:37.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.897+0000 7fd2f0625640 1 Processor -- start 2024-03-29T10:40:37.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.897+0000 7fd2f0625640 1 -- start start 2024-03-29T10:40:37.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.897+0000 7fd2f0625640 1 librados: setting wanted keys 2024-03-29T10:40:37.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.897+0000 7fd2f0625640 1 librados: calling monclient init 2024-03-29T10:40:37.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.897+0000 7fd2f0625640 10 monclient: init 2024-03-29T10:40:37.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.898+0000 7fd2f0625640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:37.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.898+0000 7fd2f0625640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:37.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.899+0000 7fd2f0625640 10 monclient(hunting): picked mon.noname-b con 0x7fd2e80a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:37.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.899+0000 7fd2f0625640 10 monclient(hunting): picked mon.noname-c con 0x7fd2e813f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:37.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.899+0000 7fd2f0625640 10 monclient(hunting): picked mon.noname-a con 0x7fd2e81457f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:37.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.899+0000 7fd2f0625640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e81da940 con 0x7fd2e81457f0 2024-03-29T10:40:37.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.899+0000 7fd2f0625640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e81da5f0 con 0x7fd2e80a9d70 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.899+0000 7fd2f0625640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e8133380 con 0x7fd2e813f650 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.899+0000 7fd2f0625640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.899+0000 7fd2f0625640 10 monclient(hunting): authenticate will time out at 903.140442s 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.899+0000 7fd2edbc0640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fd2e813f650 0x7fd2e81337d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:53656/0 (socket says 172.21.15.114:53656) 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.899+0000 7fd2edbc0640 1 -- 172.21.15.114:0/2213380225 learned_addr learned my addr 172.21.15.114:0/2213380225 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.900+0000 7fd2deffd640 10 client.?.objecter ms_handle_connect 0x7fd2e813f650 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.900+0000 7fd2deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.900+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058351366 0 0) 0x7fd2e8133380 con 0x7fd2e813f650 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.900+0000 7fd2deffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.900+0000 7fd2deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.900+0000 7fd2deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.900+0000 7fd2deffd640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:40:37.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.900+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2cc004eb0 con 0x7fd2e813f650 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.900+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 730021870 0 0) 0x7fd2cc004eb0 con 0x7fd2e813f650 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.900+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2e8133380 con 0x7fd2e813f650 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd2d8001a50 con 0x7fd2e813f650 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 10 monclient(hunting): dump: 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:37.895 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 10 client.?.objecter ms_handle_connect 0x7fd2e81457f0 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572514328 0 0) 0x7fd2e81da940 con 0x7fd2e81457f0 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2cc004eb0 con 0x7fd2e81457f0 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1002413771 0 0) 0x7fd2e8133380 con 0x7fd2e813f650 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.901+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 >> v1:172.21.15.176:6789/0 conn(0x7fd2e80a9d70 legacy=0x7fd2e81dae60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:37.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 >> v1:172.21.15.114:6789/0 conn(0x7fd2e81457f0 legacy=0x7fd2e81e1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2deffd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2deffd640 1 monclient: found mon.c 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e8133510 con 0x7fd2e813f650 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2deffd640 10 monclient: _finish_auth 0 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2deffd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2f0625640 5 monclient: authenticate success, global_id 4280 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2f0625640 10 monclient: _renew_subs 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2f0625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2f0625640 1 -- 172.21.15.114:0/2213380225 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e8133ee0 con 0x7fd2e813f650 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2f0625640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:37.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2f0625640 10 monclient: _renew_subs 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2f0625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2f0625640 1 -- 172.21.15.114:0/2213380225 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e8134410 con 0x7fd2e813f650 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.902+0000 7fd2f0625640 1 librados: init done 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.903+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2d80038a0 con 0x7fd2e813f650 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.903+0000 7fd2deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.903+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd2d8003c70 con 0x7fd2e813f650 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.903+0000 7fd2deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.903+0000 7fd2deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.903+0000 7fd2deffd640 10 monclient: dump: 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:37.898 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.903+0000 7fd2ecbbe640 4 set_mon_vals no callback set 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.905+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fd2d8003dd0 con 0x7fd2e813f650 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.905+0000 7fd2f0625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.905+0000 7fd2f0625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.905+0000 7fd2f0625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.905+0000 7fd2f0625640 1 -- 172.21.15.114:0/2213380225 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d0005080 con 0x7fd2e813f650 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.905+0000 7fd2ee3c1640 10 monclient: get_auth_request con 0x7fd2cc06dad0 auth_method 0 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.905+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fd2d8084b60 con 0x7fd2e813f650 2024-03-29T10:40:37.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.905+0000 7fd2deffd640 10 client.4280.objecter ms_dispatch 0x7fd2e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:37.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.905+0000 7fd2deffd640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:37.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.905+0000 7fd2deffd640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:37.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.906+0000 7fd2deffd640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T10:40:37.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.906+0000 7fd2deffd640 10 client.4280.objecter ms_handle_connect 0x7fd2cc06dad0 2024-03-29T10:40:37.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.913+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2d8050440 con 0x7fd2e813f650 2024-03-29T10:40:37.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.913+0000 7fd2deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:37.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:37.913+0000 7fd2deffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:38.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.242+0000 7fd2f0625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:38.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.242+0000 7fd2f0625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:38.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.242+0000 7fd2f0625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:38.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.242+0000 7fd2f0625640 1 -- 172.21.15.114:0/2213380225 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2d0005330 con 0x7fd2e813f650 2024-03-29T10:40:38.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.242+0000 7fd2deffd640 1 -- 172.21.15.114:0/2213380225 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fd2d8052370 con 0x7fd2e813f650 2024-03-29T10:40:38.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.242+0000 7fd2deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:38.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.242+0000 7fd2deffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:38.227 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:40:38.228 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:38.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.246+0000 7fd2f0625640 10 librados: watch_flush enter 2024-03-29T10:40:38.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.246+0000 7fd2f0625640 10 librados: watch_flush exit 2024-03-29T10:40:38.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.246+0000 7fd2f0625640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T10:40:38.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.246+0000 7fd2f0625640 10 client.4280.objecter successfully canceled tick 2024-03-29T10:40:38.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.246+0000 7fd2f0625640 1 -- 172.21.15.114:0/2213380225 >> v1:172.21.15.114:6816/775044307 conn(0x7fd2cc06dad0 legacy=0x7fd2cc06ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:38.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.246+0000 7fd2f0625640 10 monclient: shutdown 2024-03-29T10:40:38.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.246+0000 7fd2f0625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:38.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.246+0000 7fd2f0625640 1 -- 172.21.15.114:0/2213380225 >> v1:172.21.15.114:6790/0 conn(0x7fd2e813f650 legacy=0x7fd2e81337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:38.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.246+0000 7fd2f0625640 1 -- 172.21.15.114:0/2213380225 shutdown_connections 2024-03-29T10:40:38.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.246+0000 7fd2f0625640 1 -- 172.21.15.114:0/2213380225 >> 172.21.15.114:0/2213380225 conn(0x7fd2e8066a00 msgr2=0x7fd2e80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:38.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.247+0000 7fd2f0625640 1 -- 172.21.15.114:0/2213380225 shutdown_connections 2024-03-29T10:40:38.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.247+0000 7fd2f0625640 1 -- 172.21.15.114:0/2213380225 wait complete. 2024-03-29T10:40:38.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:38.247+0000 7fd2f0625640 1 librados: shutdown 2024-03-29T10:40:38.245 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:45.052 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:45.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.213+0000 7f6848d23640 10 monclient: get_monmap_and_config 2024-03-29T10:40:45.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.213+0000 7f6848d23640 10 monclient: build_initial_monmap 2024-03-29T10:40:45.198 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.213+0000 7f6848d23640 10 monclient: monmap: 2024-03-29T10:40:45.198 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:45.198 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:45.198 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:40:45.214817+0000 2024-03-29T10:40:45.198 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:40:45.214817+0000 2024-03-29T10:40:45.198 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:45.198 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:45.198 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:45.198 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:40:45.198 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:40:45.198 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:45.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.214+0000 7f6848d23640 1 Processor -- start 2024-03-29T10:40:45.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.214+0000 7f6848d23640 1 -- start start 2024-03-29T10:40:45.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.214+0000 7f6848d23640 10 monclient: init 2024-03-29T10:40:45.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.215+0000 7f6848d23640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:45.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.215+0000 7f6848d23640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:45.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.215+0000 7f6848d23640 10 monclient(hunting): picked mon.noname-c con 0x7f68441457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:45.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.215+0000 7f6848d23640 10 monclient(hunting): picked mon.noname-a con 0x7f684413f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:45.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.215+0000 7f6848d23640 10 monclient(hunting): picked mon.noname-b con 0x7f68440a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:45.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.215+0000 7f6848d23640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68440aaef0 con 0x7f684413f650 2024-03-29T10:40:45.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.215+0000 7f6848d23640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6844145bd0 con 0x7f68441457f0 2024-03-29T10:40:45.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.215+0000 7f6848d23640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6844145eb0 con 0x7f68440a9d70 2024-03-29T10:40:45.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.215+0000 7f6848d23640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:45.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.215+0000 7f6848d23640 10 monclient(hunting): authenticate will time out at 910.456823s 2024-03-29T10:40:45.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.218+0000 7f6842d9d640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f68440a9d70 0x7f684414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:40392/0 (socket says 172.21.15.114:40392) 2024-03-29T10:40:45.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.218+0000 7f6842d9d640 1 -- 172.21.15.114:0/1892199956 learned_addr learned my addr 172.21.15.114:0/1892199956 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:45.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 1 -- 172.21.15.114:0/1892199956 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744199034 0 0) 0x7f6844145eb0 con 0x7f68440a9d70 2024-03-29T10:40:45.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:45.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:45.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:45.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:40:45.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 1 -- 172.21.15.114:0/1892199956 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f682c003520 con 0x7f68440a9d70 2024-03-29T10:40:45.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 1 -- 172.21.15.114:0/1892199956 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318643261 0 0) 0x7f68440aaef0 con 0x7f684413f650 2024-03-29T10:40:45.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:45.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 1 -- 172.21.15.114:0/1892199956 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6844145eb0 con 0x7f684413f650 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 1 -- 172.21.15.114:0/1892199956 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 614728529 0 0) 0x7f682c003520 con 0x7f68440a9d70 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 1 -- 172.21.15.114:0/1892199956 >> v1:172.21.15.176:6789/0 conn(0x7f68441457f0 legacy=0x7f684413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 1 -- 172.21.15.114:0/1892199956 >> v1:172.21.15.114:6789/0 conn(0x7f684413f650 legacy=0x7f68440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient: _finish_hunting 0 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 1 monclient: found mon.noname-b 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 1 -- 172.21.15.114:0/1892199956 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6844146340 con 0x7f68440a9d70 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient: _finish_auth 0 2024-03-29T10:40:45.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient: _check_auth_tickets 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 1 -- 172.21.15.114:0/1892199956 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f683c001a10 con 0x7f68440a9d70 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f684159a640 10 monclient: dump: 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:45.208 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f6848d23640 5 monclient: authenticate success, global_id 4283 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.219+0000 7f6848d23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.220+0000 7f684159a640 1 -- 172.21.15.114:0/1892199956 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f683c001b70 con 0x7f68440a9d70 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.220+0000 7f684159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.220+0000 7f684159a640 1 -- 172.21.15.114:0/1892199956 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f683c001f80 con 0x7f68440a9d70 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.220+0000 7f684159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.220+0000 7f684159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.220+0000 7f684159a640 10 monclient: dump: 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:45.209 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:45.210 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:45.210 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:45.210 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:45.210 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:45.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.220+0000 7f6848d23640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:45.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.220+0000 7f6848d23640 4 set_mon_vals no callback set 2024-03-29T10:40:45.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.222+0000 7f6848d23640 10 monclient: shutdown 2024-03-29T10:40:45.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.222+0000 7f6848d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:45.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.222+0000 7f6848d23640 1 -- 172.21.15.114:0/1892199956 >> v1:172.21.15.114:6790/0 conn(0x7f68440a9d70 legacy=0x7f684414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:45.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.222+0000 7f6848d23640 1 -- 172.21.15.114:0/1892199956 shutdown_connections 2024-03-29T10:40:45.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.222+0000 7f6848d23640 1 -- 172.21.15.114:0/1892199956 >> 172.21.15.114:0/1892199956 conn(0x7f6844066a00 msgr2=0x7f6844066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:45.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.222+0000 7f6848d23640 1 -- 172.21.15.114:0/1892199956 shutdown_connections 2024-03-29T10:40:45.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.223+0000 7f6848d23640 1 -- 172.21.15.114:0/1892199956 wait complete. 2024-03-29T10:40:45.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.224+0000 7f6848d23640 10 monclient: build_initial_monmap 2024-03-29T10:40:45.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.224+0000 7f6848d23640 10 monclient: monmap: 2024-03-29T10:40:45.213 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:45.213 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.224+0000 7f6848d23640 1 librados: starting msgr at 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.224+0000 7f6848d23640 1 librados: starting objecter 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.224+0000 7f6848d23640 1 Processor -- start 2024-03-29T10:40:45.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.224+0000 7f6848d23640 1 -- start start 2024-03-29T10:40:45.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.224+0000 7f6848d23640 1 librados: setting wanted keys 2024-03-29T10:40:45.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.224+0000 7f6848d23640 1 librados: calling monclient init 2024-03-29T10:40:45.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.224+0000 7f6848d23640 10 monclient: init 2024-03-29T10:40:45.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.225+0000 7f6848d23640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:45.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.225+0000 7f6848d23640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:45.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.225+0000 7f6848d23640 10 monclient(hunting): picked mon.noname-b con 0x7f68440a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:45.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.225+0000 7f6848d23640 10 monclient(hunting): picked mon.noname-a con 0x7f684413f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:45.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.225+0000 7f6848d23640 10 monclient(hunting): picked mon.noname-c con 0x7f68441457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:45.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.225+0000 7f6848d23640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68441dbba0 con 0x7f684413f650 2024-03-29T10:40:45.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.225+0000 7f6848d23640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684406f7d0 con 0x7f68440a9d70 2024-03-29T10:40:45.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.225+0000 7f6848d23640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6844142de0 con 0x7f68441457f0 2024-03-29T10:40:45.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.225+0000 7f6848d23640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:45.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.225+0000 7f6848d23640 10 monclient(hunting): authenticate will time out at 910.466852s 2024-03-29T10:40:45.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.226+0000 7f6842d9d640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f68441457f0 0x7f6844137860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:50746/0 (socket says 172.21.15.114:50746) 2024-03-29T10:40:45.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.226+0000 7f6842d9d640 1 -- 172.21.15.114:0/2360404840 learned_addr learned my addr 172.21.15.114:0/2360404840 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.229+0000 7f6832ffd640 10 client.?.objecter ms_handle_connect 0x7f68441457f0 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991351939 0 0) 0x7f6844142de0 con 0x7f68441457f0 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6824004eb0 con 0x7f68441457f0 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 10 client.?.objecter ms_handle_connect 0x7f68440a9d70 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2128930815 0 0) 0x7f6824004eb0 con 0x7f68441457f0 2024-03-29T10:40:45.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6844142de0 con 0x7f68441457f0 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f683c001f80 con 0x7f68441457f0 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.230+0000 7f6832ffd640 10 monclient(hunting): dump: 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:45.222 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2308564012 0 0) 0x7f6844142de0 con 0x7f68441457f0 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 >> v1:172.21.15.176:6789/0 conn(0x7f68440a9d70 legacy=0x7f684413fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 >> v1:172.21.15.114:6789/0 conn(0x7f684413f650 legacy=0x7f68441dc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 1 monclient: found mon.c 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68441db730 con 0x7f68441457f0 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 10 monclient: _finish_auth 0 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.231+0000 7f6832ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:45.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.232+0000 7f6848d23640 5 monclient: authenticate success, global_id 4286 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.232+0000 7f6848d23640 10 monclient: _renew_subs 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.232+0000 7f6848d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.232+0000 7f6848d23640 1 -- 172.21.15.114:0/2360404840 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6844137f70 con 0x7f68441457f0 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.232+0000 7f6848d23640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.232+0000 7f6848d23640 10 monclient: _renew_subs 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.232+0000 7f6848d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.232+0000 7f6848d23640 1 -- 172.21.15.114:0/2360404840 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6844142490 con 0x7f68441457f0 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.232+0000 7f6848d23640 1 librados: init done 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.233+0000 7f6848d23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.234+0000 7f6848d23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.234+0000 7f6848d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.234+0000 7f6848d23640 1 -- 172.21.15.114:0/2360404840 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6844075f10 con 0x7f68441457f0 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.235+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f683c004e10 con 0x7f68441457f0 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.235+0000 7f6832ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:45.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.235+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f683c0051e0 con 0x7f68441457f0 2024-03-29T10:40:45.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.235+0000 7f6832ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:45.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.235+0000 7f6832ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:45.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.235+0000 7f6832ffd640 10 monclient: dump: 2024-03-29T10:40:45.225 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:45.225 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:45.225 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:45.225 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:45.225 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:45.225 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:45.226 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:45.226 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:45.226 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:45.226 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:45.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.235+0000 7f6840d99640 4 set_mon_vals no callback set 2024-03-29T10:40:45.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.236+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f683c005400 con 0x7f68441457f0 2024-03-29T10:40:45.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.236+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f683c084fc0 con 0x7f68441457f0 2024-03-29T10:40:45.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.236+0000 7f6832ffd640 10 client.4286.objecter ms_dispatch 0x7f6844000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:45.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.236+0000 7f6832ffd640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:45.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.236+0000 7f6832ffd640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:45.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.236+0000 7f6832ffd640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-29T10:40:45.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.236+0000 7f684259c640 10 monclient: get_auth_request con 0x7f682406dca0 auth_method 0 2024-03-29T10:40:45.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.237+0000 7f6832ffd640 10 client.4286.objecter ms_handle_connect 0x7f682406dca0 2024-03-29T10:40:45.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.241+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f683c0508a0 con 0x7f68441457f0 2024-03-29T10:40:45.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.241+0000 7f6832ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:45.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.241+0000 7f6832ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:45.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.568+0000 7f6848d23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:45.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.568+0000 7f6848d23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:45.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.568+0000 7f6848d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:45.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.568+0000 7f6848d23640 1 -- 172.21.15.114:0/2360404840 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68440b1810 con 0x7f68441457f0 2024-03-29T10:40:45.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.571+0000 7f6832ffd640 1 -- 172.21.15.114:0/2360404840 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f683c0527d0 con 0x7f68441457f0 2024-03-29T10:40:45.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.571+0000 7f6832ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:45.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.571+0000 7f6832ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:45.555 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:40:45.555 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:45.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.575+0000 7f6848d23640 10 librados: watch_flush enter 2024-03-29T10:40:45.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.575+0000 7f6848d23640 10 librados: watch_flush exit 2024-03-29T10:40:45.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.575+0000 7f6848d23640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-29T10:40:45.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.575+0000 7f6848d23640 10 client.4286.objecter successfully canceled tick 2024-03-29T10:40:45.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.575+0000 7f6848d23640 1 -- 172.21.15.114:0/2360404840 >> v1:172.21.15.114:6816/775044307 conn(0x7f682406dca0 legacy=0x7f6824070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:45.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.575+0000 7f6848d23640 10 monclient: shutdown 2024-03-29T10:40:45.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.575+0000 7f6848d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:45.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.575+0000 7f6848d23640 1 -- 172.21.15.114:0/2360404840 >> v1:172.21.15.114:6790/0 conn(0x7f68441457f0 legacy=0x7f6844137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:45.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.575+0000 7f6848d23640 1 -- 172.21.15.114:0/2360404840 shutdown_connections 2024-03-29T10:40:45.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.575+0000 7f6848d23640 1 -- 172.21.15.114:0/2360404840 >> 172.21.15.114:0/2360404840 conn(0x7f6844066a00 msgr2=0x7f6844066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:45.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.576+0000 7f6848d23640 1 -- 172.21.15.114:0/2360404840 shutdown_connections 2024-03-29T10:40:45.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.577+0000 7f6848d23640 1 -- 172.21.15.114:0/2360404840 wait complete. 2024-03-29T10:40:45.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:45.577+0000 7f6848d23640 1 librados: shutdown 2024-03-29T10:40:45.578 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:52.385 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:52.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.554+0000 7f8a41f1c640 10 monclient: get_monmap_and_config 2024-03-29T10:40:52.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.554+0000 7f8a41f1c640 10 monclient: build_initial_monmap 2024-03-29T10:40:52.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.554+0000 7f8a41f1c640 10 monclient: monmap: 2024-03-29T10:40:52.538 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:52.538 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:52.538 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:40:52.555643+0000 2024-03-29T10:40:52.539 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:40:52.555643+0000 2024-03-29T10:40:52.539 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:52.539 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:52.539 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:52.539 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:40:52.539 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:40:52.539 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:52.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.557+0000 7f8a41f1c640 1 Processor -- start 2024-03-29T10:40:52.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.558+0000 7f8a41f1c640 1 -- start start 2024-03-29T10:40:52.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.558+0000 7f8a41f1c640 10 monclient: init 2024-03-29T10:40:52.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.558+0000 7f8a41f1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:52.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.558+0000 7f8a41f1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:52.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.561+0000 7f8a41f1c640 10 monclient(hunting): picked mon.noname-c con 0x7f8a3c0b1f60 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:52.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.561+0000 7f8a41f1c640 10 monclient(hunting): picked mon.noname-b con 0x7f8a3c071eb0 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:52.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.564+0000 7f8a41f1c640 10 monclient(hunting): picked mon.noname-a con 0x7f8a3c143d10 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:52.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.564+0000 7f8a41f1c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c06ef80 con 0x7f8a3c143d10 2024-03-29T10:40:52.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.564+0000 7f8a41f1c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c06f260 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.564+0000 7f8a41f1c640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c0b40c0 con 0x7f8a3c071eb0 2024-03-29T10:40:52.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.564+0000 7f8a41f1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:52.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.564+0000 7f8a41f1c640 10 monclient(hunting): authenticate will time out at 917.805086s 2024-03-29T10:40:52.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.565+0000 7f8a3b7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f8a3c0b1f60 0x7f8a3c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:53051/0 (socket says 172.21.15.114:53051) 2024-03-29T10:40:52.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.565+0000 7f8a3b7fe640 1 -- 172.21.15.114:0/4062300982 learned_addr learned my addr 172.21.15.114:0/4062300982 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:52.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.567+0000 7f8a3a7fc640 1 -- 172.21.15.114:0/4062300982 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62456452 0 0) 0x7f8a3c0b40c0 con 0x7f8a3c071eb0 2024-03-29T10:40:52.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.567+0000 7f8a3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:52.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.567+0000 7f8a3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:52.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.567+0000 7f8a3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:52.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.567+0000 7f8a3a7fc640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:40:52.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.567+0000 7f8a3a7fc640 1 -- 172.21.15.114:0/4062300982 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a24003520 con 0x7f8a3c071eb0 2024-03-29T10:40:52.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.568+0000 7f8a3a7fc640 1 -- 172.21.15.114:0/4062300982 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3516738720 0 0) 0x7f8a24003520 con 0x7f8a3c071eb0 2024-03-29T10:40:52.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.568+0000 7f8a3a7fc640 1 -- 172.21.15.114:0/4062300982 >> v1:172.21.15.176:6789/0 conn(0x7f8a3c0b1f60 legacy=0x7f8a3c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:52.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.568+0000 7f8a3a7fc640 1 -- 172.21.15.114:0/4062300982 >> v1:172.21.15.114:6789/0 conn(0x7f8a3c143d10 legacy=0x7f8a3c146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.568+0000 7f8a3a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.568+0000 7f8a3a7fc640 1 monclient: found mon.noname-b 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.569+0000 7f8a3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.569+0000 7f8a3a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.569+0000 7f8a3a7fc640 1 -- 172.21.15.114:0/4062300982 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c155d70 con 0x7f8a3c071eb0 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.569+0000 7f8a3a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.569+0000 7f8a3a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.569+0000 7f8a3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.569+0000 7f8a3a7fc640 1 -- 172.21.15.114:0/4062300982 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8a30001690 con 0x7f8a3c071eb0 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.569+0000 7f8a3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.569+0000 7f8a3a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:52.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.569+0000 7f8a3a7fc640 10 monclient: dump: 2024-03-29T10:40:52.555 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:52.555 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:52.555 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:52.555 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:52.555 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:52.555 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:52.555 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:52.555 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:52.555 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:52.555 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:52.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.572+0000 7f8a41f1c640 5 monclient: authenticate success, global_id 4289 2024-03-29T10:40:52.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.572+0000 7f8a41f1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.574+0000 7f8a3a7fc640 1 -- 172.21.15.114:0/4062300982 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a30001860 con 0x7f8a3c071eb0 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.574+0000 7f8a3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.574+0000 7f8a3a7fc640 1 -- 172.21.15.114:0/4062300982 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8a30001cb0 con 0x7f8a3c071eb0 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.574+0000 7f8a3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.574+0000 7f8a3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.574+0000 7f8a3a7fc640 10 monclient: dump: 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:52.561 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:52.562 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:52.562 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:52.562 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:52.562 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:52.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.574+0000 7f8a41f1c640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:52.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.574+0000 7f8a41f1c640 4 set_mon_vals no callback set 2024-03-29T10:40:52.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.575+0000 7f8a41f1c640 10 monclient: shutdown 2024-03-29T10:40:52.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.575+0000 7f8a41f1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:52.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.575+0000 7f8a41f1c640 1 -- 172.21.15.114:0/4062300982 >> v1:172.21.15.114:6790/0 conn(0x7f8a3c071eb0 legacy=0x7f8a3c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:52.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.576+0000 7f8a41f1c640 1 -- 172.21.15.114:0/4062300982 shutdown_connections 2024-03-29T10:40:52.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.576+0000 7f8a41f1c640 1 -- 172.21.15.114:0/4062300982 >> 172.21.15.114:0/4062300982 conn(0x7f8a3c066a00 msgr2=0x7f8a3c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:52.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.577+0000 7f8a41f1c640 1 -- 172.21.15.114:0/4062300982 shutdown_connections 2024-03-29T10:40:52.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.578+0000 7f8a41f1c640 1 -- 172.21.15.114:0/4062300982 wait complete. 2024-03-29T10:40:52.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.579+0000 7f8a41f1c640 10 monclient: build_initial_monmap 2024-03-29T10:40:52.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.579+0000 7f8a41f1c640 10 monclient: monmap: 2024-03-29T10:40:52.564 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:52.564 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:52.564 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:40:52.564 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:40:52.564 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:52.564 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:52.564 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:52.565 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:40:52.565 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:40:52.565 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:52.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.579+0000 7f8a41f1c640 1 librados: starting msgr at 2024-03-29T10:40:52.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.579+0000 7f8a41f1c640 1 librados: starting objecter 2024-03-29T10:40:52.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.580+0000 7f8a41f1c640 1 Processor -- start 2024-03-29T10:40:52.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.580+0000 7f8a41f1c640 1 -- start start 2024-03-29T10:40:52.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.580+0000 7f8a41f1c640 1 librados: setting wanted keys 2024-03-29T10:40:52.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.580+0000 7f8a41f1c640 1 librados: calling monclient init 2024-03-29T10:40:52.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.580+0000 7f8a41f1c640 10 monclient: init 2024-03-29T10:40:52.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.581+0000 7f8a41f1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:52.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.581+0000 7f8a41f1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:52.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.581+0000 7f8a41f1c640 10 monclient(hunting): picked mon.noname-b con 0x7f8a3c0b1f60 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:52.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.581+0000 7f8a41f1c640 10 monclient(hunting): picked mon.noname-a con 0x7f8a3c143d10 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:52.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.581+0000 7f8a41f1c640 10 monclient(hunting): picked mon.noname-c con 0x7f8a3c1478d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:52.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.581+0000 7f8a41f1c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c1dfb30 con 0x7f8a3c143d10 2024-03-29T10:40:52.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.581+0000 7f8a41f1c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c14b420 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.581+0000 7f8a41f1c640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c14b700 con 0x7f8a3c1478d0 2024-03-29T10:40:52.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.581+0000 7f8a41f1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:52.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.581+0000 7f8a41f1c640 10 monclient(hunting): authenticate will time out at 917.822766s 2024-03-29T10:40:52.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.585+0000 7f8a3b7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f8a3c0b1f60 0x7f8a3c1df280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:53177/0 (socket says 172.21.15.114:53177) 2024-03-29T10:40:52.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.585+0000 7f8a3b7fe640 1 -- 172.21.15.114:0/1258024303 learned_addr learned my addr 172.21.15.114:0/1258024303 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:52.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.585+0000 7f8a23fff640 10 client.?.objecter ms_handle_connect 0x7f8a3c0b1f60 2024-03-29T10:40:52.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.585+0000 7f8a23fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:52.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.588+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307822757 0 0) 0x7f8a3c14b420 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.588+0000 7f8a23fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:52.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.588+0000 7f8a23fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:52.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.588+0000 7f8a23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:52.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.588+0000 7f8a23fff640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:40:52.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.588+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a28004eb0 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.588+0000 7f8a23fff640 10 client.?.objecter ms_handle_connect 0x7f8a3c143d10 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.588+0000 7f8a23fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.588+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399972863 0 0) 0x7f8a3c1dfb30 con 0x7f8a3c143d10 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a3c14b420 con 0x7f8a3c143d10 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 153261490 0 0) 0x7f8a28004eb0 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a3c1dfb30 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8a34001cb0 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:52.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:52.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 10 monclient(hunting): dump: 2024-03-29T10:40:52.575 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:52.575 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:52.575 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:52.575 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:52.575 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:52.575 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:52.575 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:52.575 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:52.575 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:52.576 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:52.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.589+0000 7f8a23fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:52.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.590+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3394420522 0 0) 0x7f8a3c14b420 con 0x7f8a3c143d10 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.590+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a28004eb0 con 0x7f8a3c143d10 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.590+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8a30001b70 con 0x7f8a3c143d10 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.590+0000 7f8a23fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.590+0000 7f8a23fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.590+0000 7f8a23fff640 10 monclient(hunting): dump: 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:52.580 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.590+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 279269067 0 0) 0x7f8a3c1dfb30 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.590+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 >> v1:172.21.15.114:6790/0 conn(0x7f8a3c1478d0 legacy=0x7f8a3c147d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.590+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 >> v1:172.21.15.114:6789/0 conn(0x7f8a3c143d10 legacy=0x7f8a3c154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.590+0000 7f8a23fff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a23fff640 1 monclient: found mon.b 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a23fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c148470 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a23fff640 10 monclient: _finish_auth 0 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a23fff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:52.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a41f1c640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a41f1c640 10 monclient: _renew_subs 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a41f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a41f1c640 1 -- 172.21.15.114:0/1258024303 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3c1e7310 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a41f1c640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a41f1c640 10 monclient: _renew_subs 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a41f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a41f1c640 1 -- 172.21.15.114:0/1258024303 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3c1e7680 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.592+0000 7f8a41f1c640 1 librados: init done 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.593+0000 7f8a41f1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.594+0000 7f8a41f1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.594+0000 7f8a41f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.594+0000 7f8a41f1c640 1 -- 172.21.15.114:0/1258024303 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3c075f10 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.593+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a34004b60 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.594+0000 7f8a23fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.594+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8a34004ef0 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.594+0000 7f8a23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.594+0000 7f8a23fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.594+0000 7f8a23fff640 10 monclient: dump: 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:52.583 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:52.584 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:52.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.594+0000 7f8a39ffb640 4 set_mon_vals no callback set 2024-03-29T10:40:52.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.595+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f8a34005110 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.595+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f8a34084e30 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.595+0000 7f8a23fff640 10 client.4270.objecter ms_dispatch 0x7f8a3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:52.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.596+0000 7f8a3affd640 10 monclient: get_auth_request con 0x7f8a2806de40 auth_method 0 2024-03-29T10:40:52.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.596+0000 7f8a23fff640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:52.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.596+0000 7f8a23fff640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:52.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.596+0000 7f8a23fff640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T10:40:52.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.596+0000 7f8a23fff640 10 client.4270.objecter ms_handle_connect 0x7f8a2806de40 2024-03-29T10:40:52.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.604+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a340507d0 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.605+0000 7f8a23fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:52.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.605+0000 7f8a23fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:52.925 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.940+0000 7f8a41f1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:52.925 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.940+0000 7f8a41f1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:52.925 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.940+0000 7f8a41f1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:40:52.925 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.940+0000 7f8a41f1c640 1 -- 172.21.15.114:0/1258024303 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a3c0b7380 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.945+0000 7f8a23fff640 1 -- 172.21.15.114:0/1258024303 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f8a34052700 con 0x7f8a3c0b1f60 2024-03-29T10:40:52.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.945+0000 7f8a23fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:52.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.945+0000 7f8a23fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:52.929 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:40:52.929 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:52.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.948+0000 7f8a41f1c640 10 librados: watch_flush enter 2024-03-29T10:40:52.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 10 librados: watch_flush exit 2024-03-29T10:40:52.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T10:40:52.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 10 client.4270.objecter successfully canceled tick 2024-03-29T10:40:52.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 1 -- 172.21.15.114:0/1258024303 >> v1:172.21.15.114:6816/775044307 conn(0x7f8a2806de40 legacy=0x7f8a28070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:52.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 10 monclient: shutdown 2024-03-29T10:40:52.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:52.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 1 -- 172.21.15.114:0/1258024303 >> v1:172.21.15.176:6789/0 conn(0x7f8a3c0b1f60 legacy=0x7f8a3c1df280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:52.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 1 -- 172.21.15.114:0/1258024303 shutdown_connections 2024-03-29T10:40:52.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 1 -- 172.21.15.114:0/1258024303 >> 172.21.15.114:0/1258024303 conn(0x7f8a3c066a00 msgr2=0x7f8a3c142a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:52.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 1 -- 172.21.15.114:0/1258024303 shutdown_connections 2024-03-29T10:40:52.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 1 -- 172.21.15.114:0/1258024303 wait complete. 2024-03-29T10:40:52.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:52.949+0000 7f8a41f1c640 1 librados: shutdown 2024-03-29T10:40:52.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:59.761 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:59.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.930+0000 7f4263ea1640 10 monclient: get_monmap_and_config 2024-03-29T10:40:59.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.930+0000 7f4263ea1640 10 monclient: build_initial_monmap 2024-03-29T10:40:59.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.930+0000 7f4263ea1640 10 monclient: monmap: 2024-03-29T10:40:59.914 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:59.915 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:59.915 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:40:59.931655+0000 2024-03-29T10:40:59.915 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:40:59.931655+0000 2024-03-29T10:40:59.915 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:59.915 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:59.915 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:59.915 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:40:59.915 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:40:59.915 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:59.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.931+0000 7f4263ea1640 1 Processor -- start 2024-03-29T10:40:59.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.932+0000 7f4263ea1640 1 -- start start 2024-03-29T10:40:59.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.932+0000 7f4263ea1640 10 monclient: init 2024-03-29T10:40:59.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.932+0000 7f4263ea1640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:59.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.932+0000 7f4263ea1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:59.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.932+0000 7f4263ea1640 10 monclient(hunting): picked mon.noname-a con 0x7f425c0b0770 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:59.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.932+0000 7f4263ea1640 10 monclient(hunting): picked mon.noname-c con 0x7f425c14a700 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:59.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.932+0000 7f4263ea1640 10 monclient(hunting): picked mon.noname-b con 0x7f425c0ac5a0 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:59.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.932+0000 7f4263ea1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c0aba70 con 0x7f425c0b0770 2024-03-29T10:40:59.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.933+0000 7f4263ea1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c0aecb0 con 0x7f425c14a700 2024-03-29T10:40:59.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.933+0000 7f4263ea1640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c0af150 con 0x7f425c0ac5a0 2024-03-29T10:40:59.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.933+0000 7f4263ea1640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:59.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.933+0000 7f4263ea1640 10 monclient(hunting): authenticate will time out at 925.174032s 2024-03-29T10:40:59.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.936+0000 7f426143c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f425c14a700 0x7f425c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:34517/0 (socket says 172.21.15.114:34517) 2024-03-29T10:40:59.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.936+0000 7f426143c640 1 -- 172.21.15.114:0/1575309831 learned_addr learned my addr 172.21.15.114:0/1575309831 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:59.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.937+0000 7f4260c3b640 1 -- 172.21.15.114:0/1575309831 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92606111 0 0) 0x7f425c0aecb0 con 0x7f425c14a700 2024-03-29T10:40:59.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.937+0000 7f4260c3b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:59.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.937+0000 7f4260c3b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:59.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.937+0000 7f4260c3b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:59.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.937+0000 7f4260c3b640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:40:59.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.937+0000 7f4260c3b640 1 -- 172.21.15.114:0/1575309831 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4248003520 con 0x7f425c14a700 2024-03-29T10:40:59.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.938+0000 7f4260c3b640 1 -- 172.21.15.114:0/1575309831 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2958212256 0 0) 0x7f4248003520 con 0x7f425c14a700 2024-03-29T10:40:59.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.938+0000 7f4260c3b640 1 -- 172.21.15.114:0/1575309831 >> v1:172.21.15.114:6790/0 conn(0x7f425c0ac5a0 legacy=0x7f425c151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:59.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 1 -- 172.21.15.114:0/1575309831 >> v1:172.21.15.114:6789/0 conn(0x7f425c0b0770 legacy=0x7f425c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:59.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 10 monclient: _finish_hunting 0 2024-03-29T10:40:59.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 1 monclient: found mon.noname-c 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 1 -- 172.21.15.114:0/1575309831 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c0ac9d0 con 0x7f425c14a700 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 10 monclient: _finish_auth 0 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 10 monclient: _check_auth_tickets 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 1 -- 172.21.15.114:0/1575309831 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f424c0016b0 con 0x7f425c14a700 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4260c3b640 10 monclient: dump: 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:59.927 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4263ea1640 5 monclient: authenticate success, global_id 4273 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.939+0000 7f4263ea1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.940+0000 7f4260c3b640 1 -- 172.21.15.114:0/1575309831 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f424c001910 con 0x7f425c14a700 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.940+0000 7f4260c3b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.940+0000 7f4263ea1640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.940+0000 7f4263ea1640 4 set_mon_vals no callback set 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.940+0000 7f4263ea1640 10 monclient: shutdown 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.941+0000 7f4263ea1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:59.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.941+0000 7f4263ea1640 1 -- 172.21.15.114:0/1575309831 >> v1:172.21.15.176:6789/0 conn(0x7f425c14a700 legacy=0x7f425c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:59.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.943+0000 7f4263ea1640 1 -- 172.21.15.114:0/1575309831 shutdown_connections 2024-03-29T10:40:59.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.943+0000 7f4263ea1640 1 -- 172.21.15.114:0/1575309831 >> 172.21.15.114:0/1575309831 conn(0x7f425c06b660 msgr2=0x7f425c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:59.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.945+0000 7f4263ea1640 1 -- 172.21.15.114:0/1575309831 shutdown_connections 2024-03-29T10:40:59.930 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.946+0000 7f4263ea1640 1 -- 172.21.15.114:0/1575309831 wait complete. 2024-03-29T10:40:59.931 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.948+0000 7f4263ea1640 10 monclient: build_initial_monmap 2024-03-29T10:40:59.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.948+0000 7f4263ea1640 10 monclient: monmap: 2024-03-29T10:40:59.933 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:40:59.933 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:59.933 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:40:59.933 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:40:59.933 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:59.933 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:59.934 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:40:59.934 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:40:59.934 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:40:59.934 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:59.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.948+0000 7f4263ea1640 1 librados: starting msgr at 2024-03-29T10:40:59.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.948+0000 7f4263ea1640 1 librados: starting objecter 2024-03-29T10:40:59.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.949+0000 7f4263ea1640 1 Processor -- start 2024-03-29T10:40:59.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 1 -- start start 2024-03-29T10:40:59.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 1 librados: setting wanted keys 2024-03-29T10:40:59.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 1 librados: calling monclient init 2024-03-29T10:40:59.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 10 monclient: init 2024-03-29T10:40:59.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:59.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:59.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 10 monclient(hunting): picked mon.noname-b con 0x7f425c0ac5a0 addr v1:172.21.15.176:6789/0 2024-03-29T10:40:59.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 10 monclient(hunting): picked mon.noname-c con 0x7f425c0b0770 addr v1:172.21.15.114:6790/0 2024-03-29T10:40:59.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 10 monclient(hunting): picked mon.noname-a con 0x7f425c14a700 addr v1:172.21.15.114:6789/0 2024-03-29T10:40:59.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c06ea70 con 0x7f425c14a700 2024-03-29T10:40:59.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c1def20 con 0x7f425c0ac5a0 2024-03-29T10:40:59.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c06e0e0 con 0x7f425c0b0770 2024-03-29T10:40:59.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:59.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.951+0000 7f4263ea1640 10 monclient(hunting): authenticate will time out at 925.192966s 2024-03-29T10:40:59.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.955+0000 7f426143c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f425c0b0770 0x7f425c06d9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:40837/0 (socket says 172.21.15.114:40837) 2024-03-29T10:40:59.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.955+0000 7f426143c640 1 -- 172.21.15.114:0/3932652096 learned_addr learned my addr 172.21.15.114:0/3932652096 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:40:59.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.955+0000 7f42527fc640 10 client.?.objecter ms_handle_connect 0x7f425c0b0770 2024-03-29T10:40:59.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.955+0000 7f42527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:59.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.956+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175582712 0 0) 0x7f425c06e0e0 con 0x7f425c0b0770 2024-03-29T10:40:59.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.957+0000 7f42527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:59.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.957+0000 7f42527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:59.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.957+0000 7f42527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:59.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.957+0000 7f42527fc640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:40:59.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.957+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4240004eb0 con 0x7f425c0b0770 2024-03-29T10:40:59.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.957+0000 7f42527fc640 10 client.?.objecter ms_handle_connect 0x7f425c0ac5a0 2024-03-29T10:40:59.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.957+0000 7f42527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:59.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.957+0000 7f42527fc640 10 client.?.objecter ms_handle_connect 0x7f425c14a700 2024-03-29T10:40:59.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.957+0000 7f42527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:59.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.962+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715454467 0 0) 0x7f425c1def20 con 0x7f425c0ac5a0 2024-03-29T10:40:59.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:59.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:59.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:59.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:40:59.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f425c06e0e0 con 0x7f425c0ac5a0 2024-03-29T10:40:59.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202507298 0 0) 0x7f425c06ea70 con 0x7f425c14a700 2024-03-29T10:40:59.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:59.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:59.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:59.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:40:59.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f425c1def20 con 0x7f425c14a700 2024-03-29T10:40:59.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3537496020 0 0) 0x7f4240004eb0 con 0x7f425c0b0770 2024-03-29T10:40:59.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.963+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f425c06ea70 con 0x7f425c0b0770 2024-03-29T10:40:59.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f424c001670 con 0x7f425c0b0770 2024-03-29T10:40:59.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:59.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:59.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient(hunting): dump: 2024-03-29T10:40:59.954 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:59.954 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:59.954 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:59.955 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:59.955 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:59.955 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:59.955 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:59.955 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:59.955 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:59.955 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:59.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:59.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2069104718 0 0) 0x7f425c06e0e0 con 0x7f425c0ac5a0 2024-03-29T10:40:59.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4240004eb0 con 0x7f425c0ac5a0 2024-03-29T10:40:59.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3761172295 0 0) 0x7f425c06ea70 con 0x7f425c0b0770 2024-03-29T10:40:59.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 >> v1:172.21.15.176:6789/0 conn(0x7f425c0ac5a0 legacy=0x7f425c1de6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:59.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 >> v1:172.21.15.114:6789/0 conn(0x7f425c14a700 legacy=0x7f425c13fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:59.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:59.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 1 monclient: found mon.c 2024-03-29T10:40:59.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:59.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:59.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c06e270 con 0x7f425c0b0770 2024-03-29T10:40:59.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:59.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:59.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:59.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f4263ea1640 5 monclient: authenticate success, global_id 4292 2024-03-29T10:40:59.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f4263ea1640 10 monclient: _renew_subs 2024-03-29T10:40:59.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f4263ea1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:59.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f4263ea1640 1 -- 172.21.15.114:0/3932652096 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425c1df470 con 0x7f425c0b0770 2024-03-29T10:40:59.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f4263ea1640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:59.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f4263ea1640 10 monclient: _renew_subs 2024-03-29T10:40:59.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f4263ea1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:59.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f4263ea1640 1 -- 172.21.15.114:0/3932652096 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f425c071b10 con 0x7f425c0b0770 2024-03-29T10:40:59.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f4263ea1640 1 librados: init done 2024-03-29T10:40:59.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f424c001bc0 con 0x7f425c0b0770 2024-03-29T10:40:59.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:59.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f424c0050d0 con 0x7f425c0b0770 2024-03-29T10:40:59.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:59.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:59.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.964+0000 7f42527fc640 10 monclient: dump: 2024-03-29T10:40:59.960 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:40:59.960 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:40:59.961 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:59.961 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:40:59.961 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:59.961 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:40:59.961 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:40:59.961 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:40:59.961 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:40:59.961 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:40:59.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.966+0000 7f4253fff640 4 set_mon_vals no callback set 2024-03-29T10:40:59.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.966+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f424c01b0a0 con 0x7f425c0b0770 2024-03-29T10:40:59.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.967+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f424c084c10 con 0x7f425c0b0770 2024-03-29T10:40:59.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.967+0000 7f42527fc640 10 client.4292.objecter ms_dispatch 0x7f425c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:59.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.967+0000 7f42527fc640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:59.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.967+0000 7f42527fc640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:59.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.967+0000 7f42527fc640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T10:40:59.964 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.967+0000 7f4261c3d640 10 monclient: get_auth_request con 0x7f424006df30 auth_method 0 2024-03-29T10:40:59.964 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.968+0000 7f42527fc640 10 client.4292.objecter ms_handle_connect 0x7f424006df30 2024-03-29T10:40:59.964 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.970+0000 7f4263ea1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:59.964 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.970+0000 7f4263ea1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:59.964 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.970+0000 7f4263ea1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:40:59.964 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.970+0000 7f4263ea1640 1 -- 172.21.15.114:0/3932652096 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4244005080 con 0x7f425c0b0770 2024-03-29T10:40:59.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.980+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f424c0504f0 con 0x7f425c0b0770 2024-03-29T10:40:59.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.980+0000 7f42527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:59.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:40:59.980+0000 7f42527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:00.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.305+0000 7f4263ea1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:00.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.305+0000 7f4263ea1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:00.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.305+0000 7f4263ea1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:00.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.305+0000 7f4263ea1640 1 -- 172.21.15.114:0/3932652096 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4244005330 con 0x7f425c0b0770 2024-03-29T10:41:00.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.308+0000 7f42527fc640 1 -- 172.21.15.114:0/3932652096 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f424c052420 con 0x7f425c0b0770 2024-03-29T10:41:00.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.308+0000 7f42527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:00.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.308+0000 7f42527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:00.293 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:41:00.293 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:00.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.312+0000 7f4263ea1640 10 librados: watch_flush enter 2024-03-29T10:41:00.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.313+0000 7f4263ea1640 10 librados: watch_flush exit 2024-03-29T10:41:00.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.313+0000 7f4263ea1640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T10:41:00.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.313+0000 7f4263ea1640 10 client.4292.objecter successfully canceled tick 2024-03-29T10:41:00.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.314+0000 7f4263ea1640 1 -- 172.21.15.114:0/3932652096 >> v1:172.21.15.114:6816/775044307 conn(0x7f424006df30 legacy=0x7f42400703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:00.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.314+0000 7f4263ea1640 10 monclient: shutdown 2024-03-29T10:41:00.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.314+0000 7f4263ea1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:00.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.314+0000 7f4263ea1640 1 -- 172.21.15.114:0/3932652096 >> v1:172.21.15.114:6790/0 conn(0x7f425c0b0770 legacy=0x7f425c06d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:00.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.315+0000 7f4263ea1640 1 -- 172.21.15.114:0/3932652096 shutdown_connections 2024-03-29T10:41:00.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.315+0000 7f4263ea1640 1 -- 172.21.15.114:0/3932652096 >> 172.21.15.114:0/3932652096 conn(0x7f425c06b660 msgr2=0x7f425c1416c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:00.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.316+0000 7f4263ea1640 1 -- 172.21.15.114:0/3932652096 shutdown_connections 2024-03-29T10:41:00.300 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.316+0000 7f4263ea1640 1 -- 172.21.15.114:0/3932652096 wait complete. 2024-03-29T10:41:00.300 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:00.316+0000 7f4263ea1640 1 librados: shutdown 2024-03-29T10:41:00.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:07.125 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:07.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.299+0000 7f5b0460f640 10 monclient: get_monmap_and_config 2024-03-29T10:41:07.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.299+0000 7f5b0460f640 10 monclient: build_initial_monmap 2024-03-29T10:41:07.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.299+0000 7f5b0460f640 10 monclient: monmap: 2024-03-29T10:41:07.287 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:41:07.300510+0000 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:41:07.300510+0000 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.300+0000 7f5b0460f640 1 Processor -- start 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.300+0000 7f5b0460f640 1 -- start start 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.300+0000 7f5b0460f640 10 monclient: init 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.301+0000 7f5b0460f640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:07.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.301+0000 7f5b0460f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:07.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.301+0000 7f5b0460f640 10 monclient(hunting): picked mon.noname-a con 0x7f5afc0a9930 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:07.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.301+0000 7f5b0460f640 10 monclient(hunting): picked mon.noname-c con 0x7f5afc149dc0 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:07.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.301+0000 7f5b0460f640 10 monclient(hunting): picked mon.noname-b con 0x7f5afc14ff40 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:07.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.301+0000 7f5b0460f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0aae60 con 0x7f5afc0a9930 2024-03-29T10:41:07.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.301+0000 7f5b0460f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0b4310 con 0x7f5afc149dc0 2024-03-29T10:41:07.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.301+0000 7f5b0460f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0afdd0 con 0x7f5afc14ff40 2024-03-29T10:41:07.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.301+0000 7f5b0460f640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:07.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.301+0000 7f5b0460f640 10 monclient(hunting): authenticate will time out at 932.542538s 2024-03-29T10:41:07.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.306+0000 7f5b02bac640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f5afc14ff40 0x7f5afc152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35275/0 (socket says 172.21.15.114:35275) 2024-03-29T10:41:07.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.306+0000 7f5b02bac640 1 -- 172.21.15.114:0/1936482406 learned_addr learned my addr 172.21.15.114:0/1936482406 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:07.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338242885 0 0) 0x7f5afc0afdd0 con 0x7f5afc14ff40 2024-03-29T10:41:07.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:07.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae8003520 con 0x7f5afc14ff40 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963039515 0 0) 0x7f5afc0aae60 con 0x7f5afc0a9930 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5afc0afdd0 con 0x7f5afc0a9930 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802144310 0 0) 0x7f5afc0b4310 con 0x7f5afc149dc0 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:07.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.309+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5afc0aae60 con 0x7f5afc149dc0 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1184360460 0 0) 0x7f5afc0afdd0 con 0x7f5afc0a9930 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 >> v1:172.21.15.114:6790/0 conn(0x7f5afc14ff40 legacy=0x7f5afc152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 >> v1:172.21.15.176:6789/0 conn(0x7f5afc149dc0 legacy=0x7f5afc14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 10 monclient: _finish_hunting 0 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 1 monclient: found mon.noname-a 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc14a950 con 0x7f5afc0a9930 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 10 monclient: _finish_auth 0 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 10 monclient: _check_auth_tickets 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:07.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f5af8001520 con 0x7f5afc0a9930 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 10 monclient: dump: 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:07.298 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b0460f640 5 monclient: authenticate success, global_id 4302 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b0460f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5af8001680 con 0x7f5afc0a9930 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 1 -- 172.21.15.114:0/1936482406 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f5af8001a90 con 0x7f5afc0a9930 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b013a9640 10 monclient: dump: 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:07.299 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:07.300 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:07.300 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:07.300 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:07.300 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:07.300 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b0460f640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:07.300 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.310+0000 7f5b0460f640 4 set_mon_vals no callback set 2024-03-29T10:41:07.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.311+0000 7f5b0460f640 10 monclient: shutdown 2024-03-29T10:41:07.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.311+0000 7f5b0460f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:07.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.311+0000 7f5b0460f640 1 -- 172.21.15.114:0/1936482406 >> v1:172.21.15.114:6789/0 conn(0x7f5afc0a9930 legacy=0x7f5afc143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:07.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.315+0000 7f5b0460f640 1 -- 172.21.15.114:0/1936482406 shutdown_connections 2024-03-29T10:41:07.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.315+0000 7f5b0460f640 1 -- 172.21.15.114:0/1936482406 >> 172.21.15.114:0/1936482406 conn(0x7f5afc0b35e0 msgr2=0x7f5afc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:07.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.315+0000 7f5b0460f640 1 -- 172.21.15.114:0/1936482406 shutdown_connections 2024-03-29T10:41:07.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.315+0000 7f5b0460f640 1 -- 172.21.15.114:0/1936482406 wait complete. 2024-03-29T10:41:07.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.316+0000 7f5b0460f640 10 monclient: build_initial_monmap 2024-03-29T10:41:07.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.316+0000 7f5b0460f640 10 monclient: monmap: 2024-03-29T10:41:07.303 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:07.303 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:07.303 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:41:07.303 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:41:07.303 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.316+0000 7f5b0460f640 1 librados: starting msgr at 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.316+0000 7f5b0460f640 1 librados: starting objecter 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.316+0000 7f5b0460f640 1 Processor -- start 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.318+0000 7f5b0460f640 1 -- start start 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.318+0000 7f5b0460f640 1 librados: setting wanted keys 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.318+0000 7f5b0460f640 1 librados: calling monclient init 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.318+0000 7f5b0460f640 10 monclient: init 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.319+0000 7f5b0460f640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:07.304 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.319+0000 7f5b0460f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:07.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.319+0000 7f5b0460f640 10 monclient(hunting): picked mon.noname-a con 0x7f5afc0a9930 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:07.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.323+0000 7f5b01baa640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f5afc149dc0 0x7f5afc1dba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35385/0 (socket says 172.21.15.114:35385) 2024-03-29T10:41:07.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.323+0000 7f5b01baa640 1 -- 172.21.15.114:0/3719803823 learned_addr learned my addr 172.21.15.114:0/3719803823 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:07.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.324+0000 7f5af2ffd640 10 client.?.objecter ms_handle_connect 0x7f5afc149dc0 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.324+0000 7f5af2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.324+0000 7f5b0460f640 10 monclient(hunting): picked mon.noname-c con 0x7f5afc149dc0 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.325+0000 7f5b0460f640 10 monclient(hunting): picked mon.noname-b con 0x7f5afc14ff40 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.325+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc144f50 con 0x7f5afc0a9930 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.325+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc147960 con 0x7f5afc14ff40 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.325+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc147af0 con 0x7f5afc149dc0 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.325+0000 7f5b0460f640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.325+0000 7f5b0460f640 10 monclient(hunting): authenticate will time out at 932.566787s 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527533802 0 0) 0x7f5afc147af0 con 0x7f5afc149dc0 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:07.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae0004eb0 con 0x7f5afc149dc0 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3334180461 0 0) 0x7f5ae0004eb0 con 0x7f5afc149dc0 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5afc147af0 con 0x7f5afc149dc0 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f5aec001930 con 0x7f5afc149dc0 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:07.312 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:07.313 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:07.313 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:07.313 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:07.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.326+0000 7f5af2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:07.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.327+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2010199970 0 0) 0x7f5afc147af0 con 0x7f5afc149dc0 2024-03-29T10:41:07.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.327+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 >> v1:172.21.15.176:6789/0 conn(0x7f5afc14ff40 legacy=0x7f5afc13a250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:07.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 >> v1:172.21.15.114:6789/0 conn(0x7f5afc0a9930 legacy=0x7f5afc13f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:07.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5af2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:07.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5af2ffd640 1 monclient: found mon.c 2024-03-29T10:41:07.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5af2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:07.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5af2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:07.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc13fa00 con 0x7f5afc149dc0 2024-03-29T10:41:07.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5af2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:07.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5af2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:07.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5af2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:07.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5b0460f640 5 monclient: authenticate success, global_id 4298 2024-03-29T10:41:07.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5b0460f640 10 monclient: _renew_subs 2024-03-29T10:41:07.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5b0460f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:07.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc1dc110 con 0x7f5afc149dc0 2024-03-29T10:41:07.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5b0460f640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:07.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5b0460f640 10 monclient: _renew_subs 2024-03-29T10:41:07.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5b0460f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:07.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc13e7c0 con 0x7f5afc149dc0 2024-03-29T10:41:07.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.332+0000 7f5b0460f640 1 librados: init done 2024-03-29T10:41:07.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.333+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5aec003610 con 0x7f5afc149dc0 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.333+0000 7f5af2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.333+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f5aec003a20 con 0x7f5afc149dc0 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.333+0000 7f5af2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.333+0000 7f5af2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.333+0000 7f5b00ba8640 4 set_mon_vals no callback set 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.333+0000 7f5af2ffd640 10 monclient: dump: 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:07.318 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:07.319 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:07.319 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:07.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.335+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f5aec003b80 con 0x7f5afc149dc0 2024-03-29T10:41:07.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.335+0000 7f5b0460f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:07.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.335+0000 7f5b0460f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:07.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.335+0000 7f5b0460f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:07.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.335+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae4005080 con 0x7f5afc149dc0 2024-03-29T10:41:07.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.336+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f5aec0548b0 con 0x7f5afc149dc0 2024-03-29T10:41:07.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.336+0000 7f5af2ffd640 10 client.4298.objecter ms_dispatch 0x7f5afc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:07.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.336+0000 7f5af2ffd640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:07.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.336+0000 7f5af2ffd640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:07.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.336+0000 7f5af2ffd640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T10:41:07.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.336+0000 7f5b023ab640 10 monclient: get_auth_request con 0x7f5ae006dba0 auth_method 0 2024-03-29T10:41:07.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.336+0000 7f5af2ffd640 10 client.4298.objecter ms_handle_connect 0x7f5ae006dba0 2024-03-29T10:41:07.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.345+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5aec050b50 con 0x7f5afc149dc0 2024-03-29T10:41:07.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.345+0000 7f5af2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:07.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.345+0000 7f5af2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:07.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.684+0000 7f5b0460f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:07.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.684+0000 7f5b0460f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:07.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.684+0000 7f5b0460f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:07.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.684+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ae4005330 con 0x7f5afc149dc0 2024-03-29T10:41:07.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.685+0000 7f5af2ffd640 1 -- 172.21.15.114:0/3719803823 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f5aec050cf0 con 0x7f5afc149dc0 2024-03-29T10:41:07.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.685+0000 7f5af2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:07.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.685+0000 7f5af2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:07.670 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:41:07.670 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:07.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.689+0000 7f5b0460f640 10 librados: watch_flush enter 2024-03-29T10:41:07.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.689+0000 7f5b0460f640 10 librados: watch_flush exit 2024-03-29T10:41:07.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.689+0000 7f5b0460f640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T10:41:07.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.689+0000 7f5b0460f640 10 client.4298.objecter successfully canceled tick 2024-03-29T10:41:07.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.689+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 >> v1:172.21.15.114:6816/775044307 conn(0x7f5ae006dba0 legacy=0x7f5ae0070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:07.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.689+0000 7f5b0460f640 10 monclient: shutdown 2024-03-29T10:41:07.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.689+0000 7f5b0460f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:07.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.689+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 >> v1:172.21.15.114:6790/0 conn(0x7f5afc149dc0 legacy=0x7f5afc1dba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:07.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.689+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 shutdown_connections 2024-03-29T10:41:07.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.689+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 >> 172.21.15.114:0/3719803823 conn(0x7f5afc0b35e0 msgr2=0x7f5afc0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:07.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.689+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 shutdown_connections 2024-03-29T10:41:07.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.690+0000 7f5b0460f640 1 -- 172.21.15.114:0/3719803823 wait complete. 2024-03-29T10:41:07.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:07.690+0000 7f5b0460f640 1 librados: shutdown 2024-03-29T10:41:07.689 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:14.496 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:14.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.666+0000 7f17f2172640 10 monclient: get_monmap_and_config 2024-03-29T10:41:14.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.666+0000 7f17f2172640 10 monclient: build_initial_monmap 2024-03-29T10:41:14.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.666+0000 7f17f2172640 10 monclient: monmap: 2024-03-29T10:41:14.650 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:14.650 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:14.650 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:41:14.666709+0000 2024-03-29T10:41:14.650 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:41:14.666709+0000 2024-03-29T10:41:14.650 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:14.650 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:14.650 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:14.650 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:41:14.650 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:41:14.650 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:14.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.668+0000 7f17f2172640 1 Processor -- start 2024-03-29T10:41:14.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.668+0000 7f17f2172640 1 -- start start 2024-03-29T10:41:14.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.668+0000 7f17f2172640 10 monclient: init 2024-03-29T10:41:14.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.669+0000 7f17f2172640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:14.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.669+0000 7f17f2172640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:14.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.669+0000 7f17f2172640 10 monclient(hunting): picked mon.noname-b con 0x7f17ec06d9d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:14.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.669+0000 7f17f2172640 10 monclient(hunting): picked mon.noname-a con 0x7f17ec13e820 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:14.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.669+0000 7f17f2172640 10 monclient(hunting): picked mon.noname-c con 0x7f17ec0af1d0 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:14.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.669+0000 7f17f2172640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ec0aed10 con 0x7f17ec13e820 2024-03-29T10:41:14.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.669+0000 7f17f2172640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ec0afd50 con 0x7f17ec0af1d0 2024-03-29T10:41:14.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.669+0000 7f17f2172640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ec0ab490 con 0x7f17ec06d9d0 2024-03-29T10:41:14.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.669+0000 7f17f2172640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:14.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.669+0000 7f17f2172640 10 monclient(hunting): authenticate will time out at 939.909803s 2024-03-29T10:41:14.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.674+0000 7f17eb7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f17ec06d9d0 0x7f17ec0b05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:37061/0 (socket says 172.21.15.114:37061) 2024-03-29T10:41:14.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.674+0000 7f17eb7fe640 1 -- 172.21.15.114:0/203902789 learned_addr learned my addr 172.21.15.114:0/203902789 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:14.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.677+0000 7f17ea7fc640 1 -- 172.21.15.114:0/203902789 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45302309 0 0) 0x7f17ec0ab490 con 0x7f17ec06d9d0 2024-03-29T10:41:14.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.678+0000 7f17ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:14.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.678+0000 7f17ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:14.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.678+0000 7f17ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:14.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.678+0000 7f17ea7fc640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:41:14.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.678+0000 7f17ea7fc640 1 -- 172.21.15.114:0/203902789 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d4003520 con 0x7f17ec06d9d0 2024-03-29T10:41:14.663 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.678+0000 7f17ea7fc640 1 -- 172.21.15.114:0/203902789 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 840066993 0 0) 0x7f17d4003520 con 0x7f17ec06d9d0 2024-03-29T10:41:14.663 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 1 -- 172.21.15.114:0/203902789 >> v1:172.21.15.176:6789/0 conn(0x7f17ec0af1d0 legacy=0x7f17ec0af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 1 -- 172.21.15.114:0/203902789 >> v1:172.21.15.114:6789/0 conn(0x7f17ec13e820 legacy=0x7f17ec13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 1 monclient: found mon.noname-b 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 1 -- 172.21.15.114:0/203902789 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec0ab770 con 0x7f17ec06d9d0 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 1 -- 172.21.15.114:0/203902789 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f17dc001690 con 0x7f17ec06d9d0 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 10 monclient: dump: 2024-03-29T10:41:14.664 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 1 -- 172.21.15.114:0/203902789 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17dc001920 con 0x7f17ec06d9d0 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.679+0000 7f17ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.680+0000 7f17ea7fc640 1 -- 172.21.15.114:0/203902789 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f17dc001cb0 con 0x7f17ec06d9d0 2024-03-29T10:41:14.665 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.680+0000 7f17ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.680+0000 7f17ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.680+0000 7f17ea7fc640 10 monclient: dump: 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.680+0000 7f17f2172640 5 monclient: authenticate success, global_id 4301 2024-03-29T10:41:14.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.680+0000 7f17f2172640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:14.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.680+0000 7f17f2172640 4 set_mon_vals no callback set 2024-03-29T10:41:14.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.681+0000 7f17f2172640 10 monclient: shutdown 2024-03-29T10:41:14.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.681+0000 7f17f2172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:14.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.681+0000 7f17f2172640 1 -- 172.21.15.114:0/203902789 >> v1:172.21.15.114:6790/0 conn(0x7f17ec06d9d0 legacy=0x7f17ec0b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:14.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.682+0000 7f17f2172640 1 -- 172.21.15.114:0/203902789 shutdown_connections 2024-03-29T10:41:14.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.682+0000 7f17f2172640 1 -- 172.21.15.114:0/203902789 >> 172.21.15.114:0/203902789 conn(0x7f17ec06b660 msgr2=0x7f17ec06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:14.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.684+0000 7f17f2172640 1 -- 172.21.15.114:0/203902789 shutdown_connections 2024-03-29T10:41:14.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.684+0000 7f17f2172640 1 -- 172.21.15.114:0/203902789 wait complete. 2024-03-29T10:41:14.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.685+0000 7f17f2172640 10 monclient: build_initial_monmap 2024-03-29T10:41:14.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.685+0000 7f17f2172640 10 monclient: monmap: 2024-03-29T10:41:14.668 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:14.669 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:14.669 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:41:14.669 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:41:14.669 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:14.669 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:14.669 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:14.669 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:41:14.669 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:41:14.669 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:14.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.686+0000 7f17f2172640 1 librados: starting msgr at 2024-03-29T10:41:14.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.686+0000 7f17f2172640 1 librados: starting objecter 2024-03-29T10:41:14.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.686+0000 7f17f2172640 1 Processor -- start 2024-03-29T10:41:14.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.686+0000 7f17f2172640 1 -- start start 2024-03-29T10:41:14.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.686+0000 7f17f2172640 1 librados: setting wanted keys 2024-03-29T10:41:14.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.686+0000 7f17f2172640 1 librados: calling monclient init 2024-03-29T10:41:14.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.686+0000 7f17f2172640 10 monclient: init 2024-03-29T10:41:14.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.687+0000 7f17f2172640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:14.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.687+0000 7f17f2172640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:14.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.687+0000 7f17f2172640 10 monclient(hunting): picked mon.noname-c con 0x7f17ec0af1d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:14.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.687+0000 7f17f2172640 10 monclient(hunting): picked mon.noname-a con 0x7f17ec13e820 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:14.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.687+0000 7f17f2172640 10 monclient(hunting): picked mon.noname-b con 0x7f17ec1381a0 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:14.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.687+0000 7f17f2172640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ec06ea20 con 0x7f17ec13e820 2024-03-29T10:41:14.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.687+0000 7f17f2172640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ec06ec50 con 0x7f17ec1381a0 2024-03-29T10:41:14.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.687+0000 7f17f2172640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ec06ef30 con 0x7f17ec0af1d0 2024-03-29T10:41:14.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.687+0000 7f17f2172640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:14.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.687+0000 7f17f2172640 10 monclient(hunting): authenticate will time out at 939.927593s 2024-03-29T10:41:14.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17eb7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f17ec0af1d0 0x7f17ec0b3da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:37171/0 (socket says 172.21.15.114:37171) 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17eb7fe640 1 -- 172.21.15.114:0/1719671123 learned_addr learned my addr 172.21.15.114:0/1719671123 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17d3fff640 10 client.?.objecter ms_handle_connect 0x7f17ec0af1d0 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668507391 0 0) 0x7f17ec06ef30 con 0x7f17ec0af1d0 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17d3fff640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d8004eb0 con 0x7f17ec0af1d0 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17d3fff640 10 client.?.objecter ms_handle_connect 0x7f17ec1381a0 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.693+0000 7f17d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.694+0000 7f17d3fff640 10 client.?.objecter ms_handle_connect 0x7f17ec13e820 2024-03-29T10:41:14.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.694+0000 7f17d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619192821 0 0) 0x7f17ec06ea20 con 0x7f17ec13e820 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d80051d0 con 0x7f17ec13e820 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221303635 0 0) 0x7f17ec06ec50 con 0x7f17ec1381a0 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ec06ea20 con 0x7f17ec1381a0 2024-03-29T10:41:14.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2219289671 0 0) 0x7f17d8004eb0 con 0x7f17ec0af1d0 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17ec06ec50 con 0x7f17ec0af1d0 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f17dc001550 con 0x7f17ec0af1d0 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): dump: 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:14.683 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:14.684 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:14.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:14.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4228313285 0 0) 0x7f17ec06ec50 con 0x7f17ec0af1d0 2024-03-29T10:41:14.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 >> v1:172.21.15.176:6789/0 conn(0x7f17ec1381a0 legacy=0x7f17ec1e6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:14.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 >> v1:172.21.15.114:6789/0 conn(0x7f17ec13e820 legacy=0x7f17ec06f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:14.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:14.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 monclient: found mon.c 2024-03-29T10:41:14.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:14.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec06f210 con 0x7f17ec0af1d0 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient: _finish_auth 0 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17f2172640 5 monclient: authenticate success, global_id 4304 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17f2172640 10 monclient: _renew_subs 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17f2172640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17f2172640 1 -- 172.21.15.114:0/1719671123 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ec06ddb0 con 0x7f17ec0af1d0 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17f2172640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17f2172640 10 monclient: _renew_subs 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17f2172640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17f2172640 1 -- 172.21.15.114:0/1719671123 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ec1e7a90 con 0x7f17ec0af1d0 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.697+0000 7f17f2172640 1 librados: init done 2024-03-29T10:41:14.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.699+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17dc001730 con 0x7f17ec0af1d0 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.699+0000 7f17d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.699+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f17dc001550 con 0x7f17ec0af1d0 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.699+0000 7f17d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.699+0000 7f17d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.699+0000 7f17d3fff640 10 monclient: dump: 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:14.686 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.700+0000 7f17e97fa640 4 set_mon_vals no callback set 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.700+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f17dc01aa60 con 0x7f17ec0af1d0 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.700+0000 7f17f2172640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.701+0000 7f17f2172640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.701+0000 7f17f2172640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.701+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f17dc0845b0 con 0x7f17ec0af1d0 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.701+0000 7f17d3fff640 10 client.4304.objecter ms_dispatch 0x7f17ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.701+0000 7f17d3fff640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.701+0000 7f17d3fff640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.701+0000 7f17d3fff640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T10:41:14.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.701+0000 7f17f2172640 1 -- 172.21.15.114:0/1719671123 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17cc005080 con 0x7f17ec0af1d0 2024-03-29T10:41:14.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.702+0000 7f17eaffd640 10 monclient: get_auth_request con 0x7f17d806e340 auth_method 0 2024-03-29T10:41:14.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.703+0000 7f17d3fff640 10 client.4304.objecter ms_handle_connect 0x7f17d806e340 2024-03-29T10:41:14.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.710+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17dc04fef0 con 0x7f17ec0af1d0 2024-03-29T10:41:14.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.710+0000 7f17d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:14.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:14.710+0000 7f17d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:15.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.027+0000 7f17f2172640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:15.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.027+0000 7f17f2172640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:15.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.027+0000 7f17f2172640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:15.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.027+0000 7f17f2172640 1 -- 172.21.15.114:0/1719671123 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17cc005330 con 0x7f17ec0af1d0 2024-03-29T10:41:15.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.028+0000 7f17d3fff640 1 -- 172.21.15.114:0/1719671123 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f17dc083c60 con 0x7f17ec0af1d0 2024-03-29T10:41:15.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.028+0000 7f17d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:15.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.028+0000 7f17d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:15.013 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:41:15.013 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:15.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.031+0000 7f17f2172640 10 librados: watch_flush enter 2024-03-29T10:41:15.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.031+0000 7f17f2172640 10 librados: watch_flush exit 2024-03-29T10:41:15.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.031+0000 7f17f2172640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T10:41:15.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.031+0000 7f17f2172640 10 client.4304.objecter successfully canceled tick 2024-03-29T10:41:15.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.032+0000 7f17f2172640 1 -- 172.21.15.114:0/1719671123 >> v1:172.21.15.114:6816/775044307 conn(0x7f17d806e340 legacy=0x7f17d8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:15.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.032+0000 7f17f2172640 10 monclient: shutdown 2024-03-29T10:41:15.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.032+0000 7f17f2172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:15.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.032+0000 7f17f2172640 1 -- 172.21.15.114:0/1719671123 >> v1:172.21.15.114:6790/0 conn(0x7f17ec0af1d0 legacy=0x7f17ec0b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:15.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.032+0000 7f17f2172640 1 -- 172.21.15.114:0/1719671123 shutdown_connections 2024-03-29T10:41:15.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.032+0000 7f17f2172640 1 -- 172.21.15.114:0/1719671123 >> 172.21.15.114:0/1719671123 conn(0x7f17ec06b660 msgr2=0x7f17ec14d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:15.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.034+0000 7f17f2172640 1 -- 172.21.15.114:0/1719671123 shutdown_connections 2024-03-29T10:41:15.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.034+0000 7f17f2172640 1 -- 172.21.15.114:0/1719671123 wait complete. 2024-03-29T10:41:15.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:15.034+0000 7f17f2172640 1 librados: shutdown 2024-03-29T10:41:15.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:21.844 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:21.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.008+0000 7fa154660640 10 monclient: get_monmap_and_config 2024-03-29T10:41:21.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.008+0000 7fa154660640 10 monclient: build_initial_monmap 2024-03-29T10:41:21.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.008+0000 7fa154660640 10 monclient: monmap: 2024-03-29T10:41:21.992 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:21.992 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:21.992 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:41:22.008988+0000 2024-03-29T10:41:21.992 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:41:22.008988+0000 2024-03-29T10:41:21.992 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:21.992 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:21.992 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:21.992 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:41:21.992 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:41:21.992 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:21.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.012+0000 7fa154660640 1 Processor -- start 2024-03-29T10:41:21.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.012+0000 7fa154660640 1 -- start start 2024-03-29T10:41:21.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.012+0000 7fa154660640 10 monclient: init 2024-03-29T10:41:21.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.013+0000 7fa154660640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:21.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.013+0000 7fa154660640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:21.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.013+0000 7fa154660640 10 monclient(hunting): picked mon.noname-b con 0x7fa14c1457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:21.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.014+0000 7fa154660640 10 monclient(hunting): picked mon.noname-c con 0x7fa14c13f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:21.999 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.017+0000 7fa154660640 10 monclient(hunting): picked mon.noname-a con 0x7fa14c0a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:22.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.018+0000 7fa151bfb640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fa14c13f650 0x7fa14c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:57389/0 (socket says 172.21.15.114:57389) 2024-03-29T10:41:22.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.018+0000 7fa151bfb640 1 -- 172.21.15.114:0/2715287435 learned_addr learned my addr 172.21.15.114:0/2715287435 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:22.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.019+0000 7fa154660640 1 -- 172.21.15.114:0/2715287435 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c0aaef0 con 0x7fa14c0a9d70 2024-03-29T10:41:22.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.019+0000 7fa154660640 1 -- 172.21.15.114:0/2715287435 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c145bd0 con 0x7fa14c13f650 2024-03-29T10:41:22.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.019+0000 7fa154660640 1 -- 172.21.15.114:0/2715287435 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c145eb0 con 0x7fa14c1457f0 2024-03-29T10:41:22.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.019+0000 7fa154660640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:22.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.019+0000 7fa154660640 10 monclient(hunting): authenticate will time out at 947.259352s 2024-03-29T10:41:22.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.019+0000 7fa1513fa640 1 -- 172.21.15.114:0/2715287435 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67360110 0 0) 0x7fa14c145eb0 con 0x7fa14c1457f0 2024-03-29T10:41:22.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.019+0000 7fa1513fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:22.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.019+0000 7fa1513fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:22.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.019+0000 7fa1513fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:22.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.019+0000 7fa1513fa640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:41:22.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.019+0000 7fa1513fa640 1 -- 172.21.15.114:0/2715287435 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa138003520 con 0x7fa14c1457f0 2024-03-29T10:41:22.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.020+0000 7fa1513fa640 1 -- 172.21.15.114:0/2715287435 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804670706 0 0) 0x7fa14c0aaef0 con 0x7fa14c0a9d70 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.020+0000 7fa1513fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.020+0000 7fa1513fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.020+0000 7fa1513fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.020+0000 7fa1513fa640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.020+0000 7fa1513fa640 1 -- 172.21.15.114:0/2715287435 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c145eb0 con 0x7fa14c0a9d70 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.020+0000 7fa1513fa640 1 -- 172.21.15.114:0/2715287435 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1515524254 0 0) 0x7fa138003520 con 0x7fa14c1457f0 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 1 -- 172.21.15.114:0/2715287435 >> v1:172.21.15.176:6789/0 conn(0x7fa14c13f650 legacy=0x7fa14c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 1 -- 172.21.15.114:0/2715287435 >> v1:172.21.15.114:6789/0 conn(0x7fa14c0a9d70 legacy=0x7fa14c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 10 monclient: _finish_hunting 0 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 1 monclient: found mon.noname-b 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:41:22.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 1 -- 172.21.15.114:0/2715287435 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c146340 con 0x7fa14c1457f0 2024-03-29T10:41:22.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 10 monclient: _finish_auth 0 2024-03-29T10:41:22.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 10 monclient: _check_auth_tickets 2024-03-29T10:41:22.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:22.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 1 -- 172.21.15.114:0/2715287435 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa13c001690 con 0x7fa14c1457f0 2024-03-29T10:41:22.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:22.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:22.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 10 monclient: dump: 2024-03-29T10:41:22.006 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:22.006 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:22.007 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:22.007 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:22.007 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:22.007 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:22.007 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:22.007 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:22.007 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:22.007 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:22.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa154660640 5 monclient: authenticate success, global_id 4307 2024-03-29T10:41:22.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa154660640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 1 -- 172.21.15.114:0/2715287435 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa13c0017f0 con 0x7fa14c1457f0 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa1513fa640 1 -- 172.21.15.114:0/2715287435 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa13c001c40 con 0x7fa14c1457f0 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa154660640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.021+0000 7fa154660640 4 set_mon_vals no callback set 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.022+0000 7fa154660640 10 monclient: shutdown 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.022+0000 7fa1513fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.022+0000 7fa1513fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.022+0000 7fa1513fa640 10 monclient: dump: 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:22.008 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:22.009 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:22.009 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:22.009 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:22.009 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:22.009 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:22.009 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:22.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.022+0000 7fa154660640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:22.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.022+0000 7fa154660640 1 -- 172.21.15.114:0/2715287435 >> v1:172.21.15.114:6790/0 conn(0x7fa14c1457f0 legacy=0x7fa14c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:22.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.023+0000 7fa154660640 1 -- 172.21.15.114:0/2715287435 shutdown_connections 2024-03-29T10:41:22.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.023+0000 7fa154660640 1 -- 172.21.15.114:0/2715287435 >> 172.21.15.114:0/2715287435 conn(0x7fa14c066a00 msgr2=0x7fa14c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:22.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.025+0000 7fa154660640 1 -- 172.21.15.114:0/2715287435 shutdown_connections 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.026+0000 7fa154660640 1 -- 172.21.15.114:0/2715287435 wait complete. 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.027+0000 7fa154660640 10 monclient: build_initial_monmap 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.027+0000 7fa154660640 10 monclient: monmap: 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:41:22.012 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:41:22.013 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:22.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.027+0000 7fa154660640 1 librados: starting msgr at 2024-03-29T10:41:22.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.027+0000 7fa154660640 1 librados: starting objecter 2024-03-29T10:41:22.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.027+0000 7fa154660640 1 Processor -- start 2024-03-29T10:41:22.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 1 -- start start 2024-03-29T10:41:22.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 1 librados: setting wanted keys 2024-03-29T10:41:22.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 1 librados: calling monclient init 2024-03-29T10:41:22.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 10 monclient: init 2024-03-29T10:41:22.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:22.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:22.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 10 monclient(hunting): picked mon.noname-b con 0x7fa14c0a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:22.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 10 monclient(hunting): picked mon.noname-c con 0x7fa14c13f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:22.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 10 monclient(hunting): picked mon.noname-a con 0x7fa14c1457f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:22.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c142b50 con 0x7fa14c1457f0 2024-03-29T10:41:22.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c06ec60 con 0x7fa14c0a9d70 2024-03-29T10:41:22.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c06ef40 con 0x7fa14c13f650 2024-03-29T10:41:22.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:22.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.028+0000 7fa154660640 10 monclient(hunting): authenticate will time out at 947.268988s 2024-03-29T10:41:22.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.033+0000 7fa1523fc640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fa14c0a9d70 0x7fa14c1446c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:57487/0 (socket says 172.21.15.114:57487) 2024-03-29T10:41:22.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.034+0000 7fa1523fc640 1 -- 172.21.15.114:0/2427639568 learned_addr learned my addr 172.21.15.114:0/2427639568 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:22.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.034+0000 7fa142ffd640 10 client.?.objecter ms_handle_connect 0x7fa14c1457f0 2024-03-29T10:41:22.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.034+0000 7fa142ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:22.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.035+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201753961 0 0) 0x7fa14c142b50 con 0x7fa14c1457f0 2024-03-29T10:41:22.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.035+0000 7fa142ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:22.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.035+0000 7fa142ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:22.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.035+0000 7fa142ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:22.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.035+0000 7fa142ffd640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:41:22.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.035+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa130004eb0 con 0x7fa14c1457f0 2024-03-29T10:41:22.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.035+0000 7fa142ffd640 10 client.?.objecter ms_handle_connect 0x7fa14c0a9d70 2024-03-29T10:41:22.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.035+0000 7fa142ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:22.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2708141510 0 0) 0x7fa130004eb0 con 0x7fa14c1457f0 2024-03-29T10:41:22.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa14c142b50 con 0x7fa14c1457f0 2024-03-29T10:41:22.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa144001c70 con 0x7fa14c1457f0 2024-03-29T10:41:22.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:22.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:22.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:22.019 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:22.019 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:22.019 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:22.019 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:22.020 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:22.020 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:22.020 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:22.020 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:22.020 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:22.020 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:22.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:22.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 10 client.?.objecter ms_handle_connect 0x7fa14c13f650 2024-03-29T10:41:22.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:22.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3209685331 0 0) 0x7fa14c142b50 con 0x7fa14c1457f0 2024-03-29T10:41:22.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 >> v1:172.21.15.114:6790/0 conn(0x7fa14c13f650 legacy=0x7fa14c142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:22.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 >> v1:172.21.15.176:6789/0 conn(0x7fa14c0a9d70 legacy=0x7fa14c1446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:22.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:22.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 1 monclient: found mon.a 2024-03-29T10:41:22.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:22.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:41:22.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c137aa0 con 0x7fa14c1457f0 2024-03-29T10:41:22.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:22.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.036+0000 7fa142ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.037+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa144003990 con 0x7fa14c1457f0 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.037+0000 7fa142ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.037+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa144003d60 con 0x7fa14c1457f0 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.037+0000 7fa142ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.037+0000 7fa142ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.037+0000 7fa142ffd640 10 monclient: dump: 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:22.023 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.037+0000 7fa150bf9640 4 set_mon_vals no callback set 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.037+0000 7fa154660640 5 monclient: authenticate success, global_id 4311 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.037+0000 7fa154660640 10 monclient: _renew_subs 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.037+0000 7fa154660640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.037+0000 7fa154660640 1 -- 172.21.15.114:0/2427639568 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14c071d10 con 0x7fa14c1457f0 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.038+0000 7fa154660640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.038+0000 7fa154660640 10 monclient: _renew_subs 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.038+0000 7fa154660640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.038+0000 7fa154660640 1 -- 172.21.15.114:0/2427639568 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14c071140 con 0x7fa14c1457f0 2024-03-29T10:41:22.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.038+0000 7fa154660640 1 librados: init done 2024-03-29T10:41:22.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.041+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fa144003ec0 con 0x7fa14c1457f0 2024-03-29T10:41:22.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.041+0000 7fa140ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:22.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.041+0000 7fa140ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:22.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.041+0000 7fa140ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:41:22.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.041+0000 7fa140ff9640 1 -- 172.21.15.114:0/2427639568 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14c075f10 con 0x7fa14c1457f0 2024-03-29T10:41:22.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.041+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fa144083c90 con 0x7fa14c1457f0 2024-03-29T10:41:22.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.041+0000 7fa142ffd640 10 client.4311.objecter ms_dispatch 0x7fa14c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:22.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.041+0000 7fa142ffd640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:22.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.041+0000 7fa142ffd640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:22.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.041+0000 7fa142ffd640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T10:41:22.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.042+0000 7fa1523fc640 10 monclient: get_auth_request con 0x7fa13006dcc0 auth_method 0 2024-03-29T10:41:22.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.043+0000 7fa142ffd640 10 client.4311.objecter ms_handle_connect 0x7fa13006dcc0 2024-03-29T10:41:22.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.050+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa14404f570 con 0x7fa14c1457f0 2024-03-29T10:41:22.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.050+0000 7fa142ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:22.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.050+0000 7fa142ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:22.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.369+0000 7fa140ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:22.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.369+0000 7fa140ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:22.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.369+0000 7fa140ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:41:22.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.369+0000 7fa140ff9640 1 -- 172.21.15.114:0/2427639568 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa14c0b3680 con 0x7fa14c1457f0 2024-03-29T10:41:22.355 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.371+0000 7fa142ffd640 1 -- 172.21.15.114:0/2427639568 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fa1440514a0 con 0x7fa14c1457f0 2024-03-29T10:41:22.355 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.371+0000 7fa142ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:22.355 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.371+0000 7fa142ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:22.360 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:41:22.360 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:22.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.381+0000 7fa140ff9640 10 librados: watch_flush enter 2024-03-29T10:41:22.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.381+0000 7fa140ff9640 10 librados: watch_flush exit 2024-03-29T10:41:22.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.381+0000 7fa140ff9640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T10:41:22.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.381+0000 7fa140ff9640 10 client.4311.objecter successfully canceled tick 2024-03-29T10:41:22.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.381+0000 7fa140ff9640 1 -- 172.21.15.114:0/2427639568 >> v1:172.21.15.114:6816/775044307 conn(0x7fa13006dcc0 legacy=0x7fa130070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:22.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.381+0000 7fa140ff9640 10 monclient: shutdown 2024-03-29T10:41:22.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.381+0000 7fa140ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:22.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.381+0000 7fa140ff9640 1 -- 172.21.15.114:0/2427639568 >> v1:172.21.15.114:6789/0 conn(0x7fa14c1457f0 legacy=0x7fa14c1e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:22.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.385+0000 7fa140ff9640 1 -- 172.21.15.114:0/2427639568 shutdown_connections 2024-03-29T10:41:22.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.385+0000 7fa140ff9640 1 -- 172.21.15.114:0/2427639568 >> 172.21.15.114:0/2427639568 conn(0x7fa14c066a00 msgr2=0x7fa14c13aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:22.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.386+0000 7fa140ff9640 1 -- 172.21.15.114:0/2427639568 shutdown_connections 2024-03-29T10:41:22.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.386+0000 7fa140ff9640 1 -- 172.21.15.114:0/2427639568 wait complete. 2024-03-29T10:41:22.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:22.386+0000 7fa140ff9640 1 librados: shutdown 2024-03-29T10:41:22.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:29.194 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:29.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.380+0000 7f4c57fa4640 10 monclient: get_monmap_and_config 2024-03-29T10:41:29.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.380+0000 7f4c57fa4640 10 monclient: build_initial_monmap 2024-03-29T10:41:29.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.380+0000 7f4c57fa4640 10 monclient: monmap: 2024-03-29T10:41:29.364 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:29.364 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:29.364 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:41:29.381651+0000 2024-03-29T10:41:29.364 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:41:29.381651+0000 2024-03-29T10:41:29.364 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:29.364 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:29.365 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:29.365 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:41:29.365 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:41:29.365 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:29.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.384+0000 7f4c57fa4640 1 Processor -- start 2024-03-29T10:41:29.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.384+0000 7f4c57fa4640 1 -- start start 2024-03-29T10:41:29.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.384+0000 7f4c57fa4640 10 monclient: init 2024-03-29T10:41:29.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.385+0000 7f4c57fa4640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:29.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.385+0000 7f4c57fa4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:29.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.385+0000 7f4c57fa4640 10 monclient(hunting): picked mon.noname-c con 0x7f4c501457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:29.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.385+0000 7f4c57fa4640 10 monclient(hunting): picked mon.noname-a con 0x7f4c5013f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:29.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.386+0000 7f4c56541640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f4c500a9d70 0x7f4c5014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:60888/0 (socket says 172.21.15.114:60888) 2024-03-29T10:41:29.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.386+0000 7f4c56541640 1 -- 172.21.15.114:0/607758688 learned_addr learned my addr 172.21.15.114:0/607758688 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:29.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.386+0000 7f4c57fa4640 10 monclient(hunting): picked mon.noname-b con 0x7f4c500a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:29.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.386+0000 7f4c57fa4640 1 -- 172.21.15.114:0/607758688 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c500aaef0 con 0x7f4c5013f650 2024-03-29T10:41:29.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.386+0000 7f4c57fa4640 1 -- 172.21.15.114:0/607758688 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c50145bd0 con 0x7f4c501457f0 2024-03-29T10:41:29.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.386+0000 7f4c57fa4640 1 -- 172.21.15.114:0/607758688 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c50145eb0 con 0x7f4c500a9d70 2024-03-29T10:41:29.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.387+0000 7f4c57fa4640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:29.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.387+0000 7f4c57fa4640 10 monclient(hunting): authenticate will time out at 954.628143s 2024-03-29T10:41:29.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.390+0000 7f4c54d3e640 1 -- 172.21.15.114:0/607758688 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691136219 0 0) 0x7f4c50145eb0 con 0x7f4c500a9d70 2024-03-29T10:41:29.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.390+0000 7f4c54d3e640 10 monclient(hunting): _negotiate 2024-03-29T10:41:29.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.390+0000 7f4c54d3e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:29.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.390+0000 7f4c54d3e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:29.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.390+0000 7f4c54d3e640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:41:29.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.390+0000 7f4c54d3e640 1 -- 172.21.15.114:0/607758688 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c3c003520 con 0x7f4c500a9d70 2024-03-29T10:41:29.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 1 -- 172.21.15.114:0/607758688 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2923387070 0 0) 0x7f4c3c003520 con 0x7f4c500a9d70 2024-03-29T10:41:29.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 1 -- 172.21.15.114:0/607758688 >> v1:172.21.15.176:6789/0 conn(0x7f4c501457f0 legacy=0x7f4c5013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:29.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 1 -- 172.21.15.114:0/607758688 >> v1:172.21.15.114:6789/0 conn(0x7f4c5013f650 legacy=0x7f4c500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:29.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 10 monclient: _finish_hunting 0 2024-03-29T10:41:29.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 1 monclient: found mon.noname-b 2024-03-29T10:41:29.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:29.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:41:29.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 1 -- 172.21.15.114:0/607758688 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c50146340 con 0x7f4c500a9d70 2024-03-29T10:41:29.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 10 monclient: _finish_auth 0 2024-03-29T10:41:29.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 10 monclient: _check_auth_tickets 2024-03-29T10:41:29.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:29.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 1 -- 172.21.15.114:0/607758688 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4c48001a10 con 0x7f4c500a9d70 2024-03-29T10:41:29.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:29.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:29.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 10 monclient: dump: 2024-03-29T10:41:29.378 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:29.378 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:29.378 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:29.378 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:29.378 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:29.378 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:29.378 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:29.378 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:29.379 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:29.379 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:29.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c57fa4640 5 monclient: authenticate success, global_id 4313 2024-03-29T10:41:29.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c57fa4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:29.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 1 -- 172.21.15.114:0/607758688 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c48001c30 con 0x7f4c500a9d70 2024-03-29T10:41:29.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:29.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 1 -- 172.21.15.114:0/607758688 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4c48001fc0 con 0x7f4c500a9d70 2024-03-29T10:41:29.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:29.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:29.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c54d3e640 10 monclient: dump: 2024-03-29T10:41:29.380 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:29.380 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:29.381 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:29.381 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:29.381 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:29.381 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:29.381 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:29.381 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:29.381 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:29.381 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:29.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c57fa4640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:29.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.393+0000 7f4c57fa4640 4 set_mon_vals no callback set 2024-03-29T10:41:29.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.394+0000 7f4c57fa4640 10 monclient: shutdown 2024-03-29T10:41:29.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.394+0000 7f4c57fa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:29.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.394+0000 7f4c57fa4640 1 -- 172.21.15.114:0/607758688 >> v1:172.21.15.114:6790/0 conn(0x7f4c500a9d70 legacy=0x7f4c5014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.397+0000 7f4c57fa4640 1 -- 172.21.15.114:0/607758688 shutdown_connections 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.397+0000 7f4c57fa4640 1 -- 172.21.15.114:0/607758688 >> 172.21.15.114:0/607758688 conn(0x7f4c50066a00 msgr2=0x7f4c50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.398+0000 7f4c57fa4640 1 -- 172.21.15.114:0/607758688 shutdown_connections 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.398+0000 7f4c57fa4640 1 -- 172.21.15.114:0/607758688 wait complete. 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.399+0000 7f4c57fa4640 10 monclient: build_initial_monmap 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.399+0000 7f4c57fa4640 10 monclient: monmap: 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:29.384 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:29.385 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:41:29.385 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:41:29.385 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:29.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.399+0000 7f4c57fa4640 1 librados: starting msgr at 2024-03-29T10:41:29.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.399+0000 7f4c57fa4640 1 librados: starting objecter 2024-03-29T10:41:29.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.399+0000 7f4c57fa4640 1 Processor -- start 2024-03-29T10:41:29.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.399+0000 7f4c57fa4640 1 -- start start 2024-03-29T10:41:29.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.399+0000 7f4c57fa4640 1 librados: setting wanted keys 2024-03-29T10:41:29.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.400+0000 7f4c57fa4640 1 librados: calling monclient init 2024-03-29T10:41:29.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.400+0000 7f4c57fa4640 10 monclient: init 2024-03-29T10:41:29.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.400+0000 7f4c57fa4640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:29.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.401+0000 7f4c57fa4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:29.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.401+0000 7f4c57fa4640 10 monclient(hunting): picked mon.noname-b con 0x7f4c500a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:29.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.403+0000 7f4c57fa4640 10 monclient(hunting): picked mon.noname-c con 0x7f4c5013f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:29.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.403+0000 7f4c55d40640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f4c500a9d70 0x7f4c501d71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:35467/0 (socket says 172.21.15.114:35467) 2024-03-29T10:41:29.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.406+0000 7f4c57fa4640 10 monclient(hunting): picked mon.noname-a con 0x7f4c501457f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:29.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.406+0000 7f4c57fa4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c50141b10 con 0x7f4c501457f0 2024-03-29T10:41:29.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.406+0000 7f4c57fa4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5013fda0 con 0x7f4c500a9d70 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.406+0000 7f4c55d40640 1 -- 172.21.15.114:0/1614607126 learned_addr learned my addr 172.21.15.114:0/1614607126 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.406+0000 7f4c57fa4640 1 -- 172.21.15.114:0/1614607126 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c50140080 con 0x7f4c5013f650 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.406+0000 7f4c57fa4640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.406+0000 7f4c57fa4640 10 monclient(hunting): authenticate will time out at 954.647923s 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 10 client.?.objecter ms_handle_connect 0x7f4c500a9d70 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 10 client.?.objecter ms_handle_connect 0x7f4c501457f0 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594102410 0 0) 0x7f4c5013fda0 con 0x7f4c500a9d70 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:41:29.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c34004eb0 con 0x7f4c500a9d70 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 10 client.?.objecter ms_handle_connect 0x7f4c5013f650 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.407+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569127281 0 0) 0x7f4c50141b10 con 0x7f4c501457f0 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.408+0000 7f4c467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.408+0000 7f4c467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.408+0000 7f4c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.408+0000 7f4c467fc640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.408+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c5013fda0 con 0x7f4c501457f0 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.408+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 691826651 0 0) 0x7f4c34004eb0 con 0x7f4c500a9d70 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.408+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c50141b10 con 0x7f4c500a9d70 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.408+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134906045 0 0) 0x7f4c50140080 con 0x7f4c5013f650 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.408+0000 7f4c467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.409+0000 7f4c467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:29.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.409+0000 7f4c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.409+0000 7f4c467fc640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.409+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c34004eb0 con 0x7f4c5013f650 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.409+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4c40001970 con 0x7f4c500a9d70 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.409+0000 7f4c467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.409+0000 7f4c467fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.409+0000 7f4c467fc640 10 monclient(hunting): dump: 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:29.399 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.409+0000 7f4c467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.410+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3897741813 0 0) 0x7f4c5013fda0 con 0x7f4c501457f0 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.410+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c50140080 con 0x7f4c501457f0 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.410+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2759837125 0 0) 0x7f4c50141b10 con 0x7f4c500a9d70 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.410+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 >> v1:172.21.15.114:6790/0 conn(0x7f4c5013f650 legacy=0x7f4c501405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.410+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 >> v1:172.21.15.114:6789/0 conn(0x7f4c501457f0 legacy=0x7f4c50142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.410+0000 7f4c467fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.410+0000 7f4c467fc640 1 monclient: found mon.b 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.410+0000 7f4c467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.410+0000 7f4c467fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:41:29.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.410+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5013b560 con 0x7f4c500a9d70 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.411+0000 7f4c467fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.411+0000 7f4c467fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.411+0000 7f4c467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.411+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c40001ad0 con 0x7f4c500a9d70 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.411+0000 7f4c57fa4640 5 monclient: authenticate success, global_id 4291 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.411+0000 7f4c57fa4640 10 monclient: _renew_subs 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.411+0000 7f4c57fa4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.411+0000 7f4c57fa4640 1 -- 172.21.15.114:0/1614607126 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5013b810 con 0x7f4c500a9d70 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.411+0000 7f4c57fa4640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.411+0000 7f4c57fa4640 10 monclient: _renew_subs 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.412+0000 7f4c57fa4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.412+0000 7f4c57fa4640 1 -- 172.21.15.114:0/1614607126 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c501d45b0 con 0x7f4c500a9d70 2024-03-29T10:41:29.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.412+0000 7f4c57fa4640 1 librados: init done 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.412+0000 7f4c467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.412+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4c400038c0 con 0x7f4c500a9d70 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.412+0000 7f4c477fe640 4 set_mon_vals no callback set 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.412+0000 7f4c467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.412+0000 7f4c467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.412+0000 7f4c467fc640 10 monclient: dump: 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:29.402 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:29.403 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:29.403 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:29.403 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:29.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.414+0000 7f4c57fa4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:29.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.414+0000 7f4c57fa4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:29.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.414+0000 7f4c57fa4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:41:29.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.414+0000 7f4c57fa4640 1 -- 172.21.15.114:0/1614607126 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c38005080 con 0x7f4c500a9d70 2024-03-29T10:41:29.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.414+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f4c40003a20 con 0x7f4c500a9d70 2024-03-29T10:41:29.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.415+0000 7f4c5553f640 10 monclient: get_auth_request con 0x7f4c3406ddf0 auth_method 0 2024-03-29T10:41:29.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.415+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f4c400849b0 con 0x7f4c500a9d70 2024-03-29T10:41:29.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.415+0000 7f4c467fc640 10 client.4291.objecter ms_dispatch 0x7f4c50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:29.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.415+0000 7f4c467fc640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:29.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.415+0000 7f4c467fc640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:29.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.415+0000 7f4c467fc640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T10:41:29.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.415+0000 7f4c467fc640 10 client.4291.objecter ms_handle_connect 0x7f4c3406ddf0 2024-03-29T10:41:29.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.423+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c40050290 con 0x7f4c500a9d70 2024-03-29T10:41:29.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.423+0000 7f4c467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:29.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.423+0000 7f4c467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:29.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.824+0000 7f4c57fa4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:29.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.824+0000 7f4c57fa4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:29.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.824+0000 7f4c57fa4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:41:29.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.824+0000 7f4c57fa4640 1 -- 172.21.15.114:0/1614607126 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c38005330 con 0x7f4c500a9d70 2024-03-29T10:41:29.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.825+0000 7f4c467fc640 1 -- 172.21.15.114:0/1614607126 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f4c400521c0 con 0x7f4c500a9d70 2024-03-29T10:41:29.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.825+0000 7f4c467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:29.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.825+0000 7f4c467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:29.809 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:41:29.809 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.829+0000 7f4c57fa4640 10 librados: watch_flush enter 2024-03-29T10:41:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.829+0000 7f4c57fa4640 10 librados: watch_flush exit 2024-03-29T10:41:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.829+0000 7f4c57fa4640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T10:41:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.829+0000 7f4c57fa4640 10 client.4291.objecter successfully canceled tick 2024-03-29T10:41:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.829+0000 7f4c57fa4640 1 -- 172.21.15.114:0/1614607126 >> v1:172.21.15.114:6816/775044307 conn(0x7f4c3406ddf0 legacy=0x7f4c340702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.829+0000 7f4c57fa4640 10 monclient: shutdown 2024-03-29T10:41:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.829+0000 7f4c57fa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.829+0000 7f4c57fa4640 1 -- 172.21.15.114:0/1614607126 >> v1:172.21.15.176:6789/0 conn(0x7f4c500a9d70 legacy=0x7f4c501d71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:29.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.830+0000 7f4c57fa4640 1 -- 172.21.15.114:0/1614607126 shutdown_connections 2024-03-29T10:41:29.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.830+0000 7f4c57fa4640 1 -- 172.21.15.114:0/1614607126 >> 172.21.15.114:0/1614607126 conn(0x7f4c50066a00 msgr2=0x7f4c500b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:29.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.830+0000 7f4c57fa4640 1 -- 172.21.15.114:0/1614607126 shutdown_connections 2024-03-29T10:41:29.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.830+0000 7f4c57fa4640 1 -- 172.21.15.114:0/1614607126 wait complete. 2024-03-29T10:41:29.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:29.831+0000 7f4c57fa4640 1 librados: shutdown 2024-03-29T10:41:29.836 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:36.644 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:36.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.813+0000 7f8aa14d9640 10 monclient: get_monmap_and_config 2024-03-29T10:41:36.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.813+0000 7f8aa14d9640 10 monclient: build_initial_monmap 2024-03-29T10:41:36.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.813+0000 7f8aa14d9640 10 monclient: monmap: 2024-03-29T10:41:36.797 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:36.797 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:36.798 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:41:36.815018+0000 2024-03-29T10:41:36.798 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:41:36.815018+0000 2024-03-29T10:41:36.798 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:36.798 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:36.798 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:36.798 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:41:36.798 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:41:36.798 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:36.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.814+0000 7f8aa14d9640 1 Processor -- start 2024-03-29T10:41:36.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.814+0000 7f8aa14d9640 1 -- start start 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.814+0000 7f8aa14d9640 10 monclient: init 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8aa14d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8aa14d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8aa14d9640 10 monclient(hunting): picked mon.noname-b con 0x7f8a9c1457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8aa14d9640 10 monclient(hunting): picked mon.noname-a con 0x7f8a9c13f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8aa14d9640 10 monclient(hunting): picked mon.noname-c con 0x7f8a9c0a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8aa14d9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c0aaef0 con 0x7f8a9c13f650 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8aa14d9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c145bd0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8a9b7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f8a9c0a9d70 0x7f8a9c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:38402/0 (socket says 172.21.15.114:38402) 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8a9b7fe640 1 -- 172.21.15.114:0/991090572 learned_addr learned my addr 172.21.15.114:0/991090572 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8aa14d9640 1 -- 172.21.15.114:0/991090572 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c145eb0 con 0x7f8a9c1457f0 2024-03-29T10:41:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8aa14d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.815+0000 7f8aa14d9640 10 monclient(hunting): authenticate will time out at 962.056846s 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973336435 0 0) 0x7f8a9c145eb0 con 0x7f8a9c1457f0 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a84003520 con 0x7f8a9c1457f0 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469212081 0 0) 0x7f8a9c145bd0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a840048a0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116341240 0 0) 0x7f8a9c0aaef0 con 0x7f8a9c13f650 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a9c145bd0 con 0x7f8a9c13f650 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2098803129 0 0) 0x7f8a84003520 con 0x7f8a9c1457f0 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 >> v1:172.21.15.176:6789/0 conn(0x7f8a9c0a9d70 legacy=0x7f8a9c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 >> v1:172.21.15.114:6789/0 conn(0x7f8a9c13f650 legacy=0x7f8a9c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 monclient: found mon.noname-b 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c146340 con 0x7f8a9c1457f0 2024-03-29T10:41:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient: _finish_auth 0 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8a940019f0 con 0x7f8a9c1457f0 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8a99ffb640 10 monclient: dump: 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:36.805 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8aa14d9640 5 monclient: authenticate success, global_id 4319 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.816+0000 7f8aa14d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.817+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a94001b50 con 0x7f8a9c1457f0 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.817+0000 7f8a99ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.817+0000 7f8aa14d9640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.817+0000 7f8aa14d9640 4 set_mon_vals no callback set 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.817+0000 7f8a99ffb640 1 -- 172.21.15.114:0/991090572 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8a94001f60 con 0x7f8a9c1457f0 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.817+0000 7f8a99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.817+0000 7f8a99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.817+0000 7f8a99ffb640 10 monclient: dump: 2024-03-29T10:41:36.806 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.817+0000 7f8aa14d9640 10 monclient: shutdown 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.817+0000 7f8aa14d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.817+0000 7f8aa14d9640 1 -- 172.21.15.114:0/991090572 >> v1:172.21.15.114:6790/0 conn(0x7f8a9c1457f0 legacy=0x7f8a9c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:36.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.818+0000 7f8aa14d9640 1 -- 172.21.15.114:0/991090572 shutdown_connections 2024-03-29T10:41:36.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.818+0000 7f8aa14d9640 1 -- 172.21.15.114:0/991090572 >> 172.21.15.114:0/991090572 conn(0x7f8a9c066a00 msgr2=0x7f8a9c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:36.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.818+0000 7f8aa14d9640 1 -- 172.21.15.114:0/991090572 shutdown_connections 2024-03-29T10:41:36.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.818+0000 7f8aa14d9640 1 -- 172.21.15.114:0/991090572 wait complete. 2024-03-29T10:41:36.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.818+0000 7f8aa14d9640 10 monclient: build_initial_monmap 2024-03-29T10:41:36.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.818+0000 7f8aa14d9640 10 monclient: monmap: 2024-03-29T10:41:36.814 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.819+0000 7f8aa14d9640 1 librados: starting msgr at 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.819+0000 7f8aa14d9640 1 librados: starting objecter 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.819+0000 7f8aa14d9640 1 Processor -- start 2024-03-29T10:41:36.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.819+0000 7f8aa14d9640 1 -- start start 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.819+0000 7f8aa14d9640 1 librados: setting wanted keys 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.819+0000 7f8aa14d9640 1 librados: calling monclient init 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.820+0000 7f8aa14d9640 10 monclient: init 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8aa14d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8aa14d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8aa14d9640 10 monclient(hunting): picked mon.noname-c con 0x7f8a9c0a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8aa14d9640 10 monclient(hunting): picked mon.noname-b con 0x7f8a9c13f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8a9affd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f8a9c0a9d70 0x7f8a9c13fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:53602/0 (socket says 172.21.15.114:53602) 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8aa14d9640 10 monclient(hunting): picked mon.noname-a con 0x7f8a9c1457f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8aa14d9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c1dbce0 con 0x7f8a9c1457f0 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8a9b7fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8a9c1457f0 0x7f8a9c1e6f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38886/0 (socket says 172.21.15.114:38886) 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8a9b7fe640 1 -- 172.21.15.114:0/338552065 learned_addr learned my addr 172.21.15.114:0/338552065 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c142d90 con 0x7f8a9c13f650 2024-03-29T10:41:36.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c1db730 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8aa14d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8aa14d9640 10 monclient(hunting): authenticate will time out at 962.062806s 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8a837fe640 10 client.?.objecter ms_handle_connect 0x7f8a9c0a9d70 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8a837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8a837fe640 10 client.?.objecter ms_handle_connect 0x7f8a9c1457f0 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.821+0000 7f8a837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387609300 0 0) 0x7f8a9c1db730 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a88004eb0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 10 client.?.objecter ms_handle_connect 0x7f8a9c13f650 2024-03-29T10:41:36.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753660309 0 0) 0x7f8a9c1dbce0 con 0x7f8a9c1457f0 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a9c1db730 con 0x7f8a9c1457f0 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.822+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2192718636 0 0) 0x7f8a88004eb0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a9c1dbce0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8a94001ea0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient(hunting): dump: 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:36.818 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753735059 0 0) 0x7f8a9c142d90 con 0x7f8a9c13f650 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a88004eb0 con 0x7f8a9c13f650 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 527055128 0 0) 0x7f8a9c1dbce0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 >> v1:172.21.15.176:6789/0 conn(0x7f8a9c13f650 legacy=0x7f8a9c1daab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 >> v1:172.21.15.114:6789/0 conn(0x7f8a9c1457f0 legacy=0x7f8a9c1e6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient: _finish_hunting 0 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 1 monclient: found mon.c 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c1db8c0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient: _finish_auth 0 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 10 monclient: _check_auth_tickets 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8a837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8aa14d9640 5 monclient: authenticate success, global_id 4322 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8aa14d9640 10 monclient: _renew_subs 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8aa14d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c1db1c0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8aa14d9640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8aa14d9640 10 monclient: _renew_subs 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8aa14d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c1426b0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.823+0000 7f8aa14d9640 1 librados: init done 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.824+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a94003c20 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.824+0000 7f8a837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.824+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8a94002100 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.824+0000 7f8a837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.824+0000 7f8a997fa640 4 set_mon_vals no callback set 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.824+0000 7f8a837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.824+0000 7f8a837fe640 10 monclient: dump: 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.824+0000 7f8aa14d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:36.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.825+0000 7f8aa14d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.825+0000 7f8aa14d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.825+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a7c005080 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.826+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f8a94003fb0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.826+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f8a94084d10 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.827+0000 7f8a837fe640 10 client.4322.objecter ms_dispatch 0x7f8a9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.827+0000 7f8a9a7fc640 10 monclient: get_auth_request con 0x7f8a8806de60 auth_method 0 2024-03-29T10:41:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.827+0000 7f8a837fe640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.827+0000 7f8a837fe640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.827+0000 7f8a837fe640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T10:41:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.827+0000 7f8a837fe640 10 client.4322.objecter ms_handle_connect 0x7f8a8806de60 2024-03-29T10:41:36.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.835+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a940505f0 con 0x7f8a9c0a9d70 2024-03-29T10:41:36.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.835+0000 7f8a837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:36.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:36.835+0000 7f8a837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:37.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.200+0000 7f8aa14d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:37.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.200+0000 7f8aa14d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:37.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.200+0000 7f8aa14d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:37.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.200+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a7c005330 con 0x7f8a9c0a9d70 2024-03-29T10:41:37.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.200+0000 7f8a837fe640 1 -- 172.21.15.114:0/338552065 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f8a94052520 con 0x7f8a9c0a9d70 2024-03-29T10:41:37.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.200+0000 7f8a837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:37.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.200+0000 7f8a837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:37.185 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:41:37.185 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:37.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.204+0000 7f8aa14d9640 10 librados: watch_flush enter 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.204+0000 7f8aa14d9640 10 librados: watch_flush exit 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.204+0000 7f8aa14d9640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.204+0000 7f8aa14d9640 10 client.4322.objecter successfully canceled tick 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.204+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 >> v1:172.21.15.114:6816/775044307 conn(0x7f8a8806de60 legacy=0x7f8a88070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.204+0000 7f8aa14d9640 10 monclient: shutdown 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.204+0000 7f8aa14d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.204+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 >> v1:172.21.15.114:6790/0 conn(0x7f8a9c0a9d70 legacy=0x7f8a9c13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.205+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 shutdown_connections 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.205+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 >> 172.21.15.114:0/338552065 conn(0x7f8a9c066a00 msgr2=0x7f8a9c0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.205+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 shutdown_connections 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.205+0000 7f8aa14d9640 1 -- 172.21.15.114:0/338552065 wait complete. 2024-03-29T10:41:37.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:37.205+0000 7f8aa14d9640 1 librados: shutdown 2024-03-29T10:41:37.203 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:44.010 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:44.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.158+0000 7f3dc437b640 10 monclient: get_monmap_and_config 2024-03-29T10:41:44.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.158+0000 7f3dc437b640 10 monclient: build_initial_monmap 2024-03-29T10:41:44.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.158+0000 7f3dc437b640 10 monclient: monmap: 2024-03-29T10:41:44.143 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:44.143 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:44.143 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:41:44.160158+0000 2024-03-29T10:41:44.143 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:41:44.160158+0000 2024-03-29T10:41:44.143 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:44.143 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:44.143 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:44.144 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:41:44.144 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:41:44.144 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:44.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.159+0000 7f3dc437b640 1 Processor -- start 2024-03-29T10:41:44.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.159+0000 7f3dc437b640 1 -- start start 2024-03-29T10:41:44.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.159+0000 7f3dc437b640 10 monclient: init 2024-03-29T10:41:44.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.160+0000 7f3dc437b640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:44.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.160+0000 7f3dc437b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:44.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.160+0000 7f3dc437b640 10 monclient(hunting): picked mon.noname-b con 0x7f3dbc1457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:44.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.160+0000 7f3dc437b640 10 monclient(hunting): picked mon.noname-c con 0x7f3dbc13f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.160+0000 7f3dc437b640 10 monclient(hunting): picked mon.noname-a con 0x7f3dbc0a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.160+0000 7f3dc437b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc0aaef0 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.160+0000 7f3dc437b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc145bd0 con 0x7f3dbc13f650 2024-03-29T10:41:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.160+0000 7f3dc437b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc145eb0 con 0x7f3dbc1457f0 2024-03-29T10:41:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.160+0000 7f3dc437b640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.160+0000 7f3dc437b640 10 monclient(hunting): authenticate will time out at 969.401933s 2024-03-29T10:41:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.161+0000 7f3dc2117640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f3dbc1457f0 0x7f3dbc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55096/0 (socket says 172.21.15.114:55096) 2024-03-29T10:41:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.161+0000 7f3dc2117640 1 -- 172.21.15.114:0/3518626620 learned_addr learned my addr 172.21.15.114:0/3518626620 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:44.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 1 -- 172.21.15.114:0/3518626620 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093461452 0 0) 0x7f3dbc145eb0 con 0x7f3dbc1457f0 2024-03-29T10:41:44.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 10 monclient(hunting): _negotiate 2024-03-29T10:41:44.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:44.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:44.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:41:44.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 1 -- 172.21.15.114:0/3518626620 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da8003520 con 0x7f3dbc1457f0 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 1 -- 172.21.15.114:0/3518626620 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458618533 0 0) 0x7f3dbc145bd0 con 0x7f3dbc13f650 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 10 monclient(hunting): _negotiate 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 1 -- 172.21.15.114:0/3518626620 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dbc145eb0 con 0x7f3dbc13f650 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 1 -- 172.21.15.114:0/3518626620 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2649082735 0 0) 0x7f3da8003520 con 0x7f3dbc1457f0 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.162+0000 7f3dc1115640 1 -- 172.21.15.114:0/3518626620 >> v1:172.21.15.176:6789/0 conn(0x7f3dbc13f650 legacy=0x7f3dbc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.163+0000 7f3dc1115640 1 -- 172.21.15.114:0/3518626620 >> v1:172.21.15.114:6789/0 conn(0x7f3dbc0a9d70 legacy=0x7f3dbc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 10 monclient: _finish_hunting 0 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 1 monclient: found mon.noname-b 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:41:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 1 -- 172.21.15.114:0/3518626620 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc146340 con 0x7f3dbc1457f0 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 10 monclient: _finish_auth 0 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 10 monclient: _check_auth_tickets 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 1 -- 172.21.15.114:0/3518626620 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3db8001690 con 0x7f3dbc1457f0 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc437b640 5 monclient: authenticate success, global_id 4325 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc437b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 10 monclient: dump: 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:44.152 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc437b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 1 -- 172.21.15.114:0/3518626620 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3db80018e0 con 0x7f3dbc1457f0 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc1115640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc437b640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.164+0000 7f3dc437b640 4 set_mon_vals no callback set 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.165+0000 7f3dc1115640 1 -- 172.21.15.114:0/3518626620 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3db8001c70 con 0x7f3dbc1457f0 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.165+0000 7f3dc1115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.165+0000 7f3dc1115640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.165+0000 7f3dc1115640 10 monclient: dump: 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.165+0000 7f3dc437b640 10 monclient: shutdown 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.165+0000 7f3dc437b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.165+0000 7f3dc437b640 1 -- 172.21.15.114:0/3518626620 >> v1:172.21.15.114:6790/0 conn(0x7f3dbc1457f0 legacy=0x7f3dbc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:44.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.165+0000 7f3dc437b640 1 -- 172.21.15.114:0/3518626620 shutdown_connections 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.165+0000 7f3dc437b640 1 -- 172.21.15.114:0/3518626620 >> 172.21.15.114:0/3518626620 conn(0x7f3dbc066a00 msgr2=0x7f3dbc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.166+0000 7f3dc437b640 1 -- 172.21.15.114:0/3518626620 shutdown_connections 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.166+0000 7f3dc437b640 1 -- 172.21.15.114:0/3518626620 wait complete. 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.166+0000 7f3dc437b640 10 monclient: build_initial_monmap 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.167+0000 7f3dc437b640 10 monclient: monmap: 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:41:44.155 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:41:44.156 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.167+0000 7f3dc437b640 1 librados: starting msgr at 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.167+0000 7f3dc437b640 1 librados: starting objecter 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.167+0000 7f3dc437b640 1 Processor -- start 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.167+0000 7f3dc437b640 1 -- start start 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.167+0000 7f3dc437b640 1 librados: setting wanted keys 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.167+0000 7f3dc437b640 1 librados: calling monclient init 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.167+0000 7f3dc437b640 10 monclient: init 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc437b640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc437b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc437b640 10 monclient(hunting): picked mon.noname-b con 0x7f3dbc0a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc437b640 10 monclient(hunting): picked mon.noname-a con 0x7f3dbc13f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc437b640 10 monclient(hunting): picked mon.noname-c con 0x7f3dbc1457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc437b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc1d7780 con 0x7f3dbc13f650 2024-03-29T10:41:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc437b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc1d7910 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc437b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc142c60 con 0x7f3dbc1457f0 2024-03-29T10:41:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc437b640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc437b640 10 monclient(hunting): authenticate will time out at 969.409614s 2024-03-29T10:41:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc2918640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f3dbc1457f0 0x7f3dbc133c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55320/0 (socket says 172.21.15.114:55320) 2024-03-29T10:41:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.168+0000 7f3dc2918640 1 -- 172.21.15.114:0/481072462 learned_addr learned my addr 172.21.15.114:0/481072462 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.169+0000 7f3db2ffd640 10 client.?.objecter ms_handle_connect 0x7f3dbc0a9d70 2024-03-29T10:41:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.169+0000 7f3db2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.169+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763832897 0 0) 0x7f3dbc1d7910 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.169+0000 7f3db2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.169+0000 7f3db2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.169+0000 7f3db2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.169+0000 7f3db2ffd640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:41:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.169+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da0004eb0 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 client.?.objecter ms_handle_connect 0x7f3dbc1457f0 2024-03-29T10:41:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203880971 0 0) 0x7f3dbc142c60 con 0x7f3dbc1457f0 2024-03-29T10:41:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dbc1d7910 con 0x7f3dbc1457f0 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 211228931 0 0) 0x7f3da0004eb0 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dbc142c60 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 53913220 0 0) 0x7f3dbc1d7910 con 0x7f3dbc1457f0 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3da0004eb0 con 0x7f3dbc1457f0 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3db8001bf0 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:44.164 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:44.165 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:44.165 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:44.165 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:44.165 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:44.165 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:44.165 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:44.165 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:44.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:44.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3db4001ba0 con 0x7f3dbc1457f0 2024-03-29T10:41:44.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:44.166 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:44.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 713385328 0 0) 0x7f3dbc142c60 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 >> v1:172.21.15.114:6790/0 conn(0x7f3dbc1457f0 legacy=0x7f3dbc133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:44.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.170+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 >> v1:172.21.15.114:6789/0 conn(0x7f3dbc13f650 legacy=0x7f3dbc136600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:44.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.171+0000 7f3db2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:44.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 1 monclient: found mon.b 2024-03-29T10:41:44.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:41:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc142df0 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3db8004b00 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3db8001d50 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3db2ffd640 10 monclient: dump: 2024-03-29T10:41:44.170 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:44.170 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:44.170 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:44.170 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:44.170 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:44.170 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:44.170 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:44.170 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:44.170 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3dc437b640 5 monclient: authenticate success, global_id 4303 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3dc437b640 10 monclient: _renew_subs 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3dc437b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3dc437b640 1 -- 172.21.15.114:0/481072462 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dbc134c50 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3dc0914640 4 set_mon_vals no callback set 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3dc437b640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3dc437b640 10 monclient: _renew_subs 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3dc437b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3dc437b640 1 -- 172.21.15.114:0/481072462 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dbc1428a0 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.172+0000 7f3dc437b640 1 librados: init done 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.174+0000 7f3dc437b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.174+0000 7f3dc437b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.174+0000 7f3dc437b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:41:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.174+0000 7f3dc437b640 1 -- 172.21.15.114:0/481072462 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dbc134ff0 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.174+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f3db801ad40 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.175+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f3db8084b50 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.175+0000 7f3db2ffd640 10 client.4303.objecter ms_dispatch 0x7f3dbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.175+0000 7f3db2ffd640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.175+0000 7f3db2ffd640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.175+0000 7f3dc1916640 10 monclient: get_auth_request con 0x7f3da006de50 auth_method 0 2024-03-29T10:41:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.175+0000 7f3db2ffd640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-29T10:41:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.175+0000 7f3db2ffd640 10 client.4303.objecter ms_handle_connect 0x7f3da006de50 2024-03-29T10:41:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.182+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3db8050430 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.182+0000 7f3db2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.182+0000 7f3db2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:44.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.505+0000 7f3dc437b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:44.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.505+0000 7f3dc437b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:44.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.505+0000 7f3dc437b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:41:44.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.505+0000 7f3dc437b640 1 -- 172.21.15.114:0/481072462 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dbc134de0 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.506+0000 7f3db2ffd640 1 -- 172.21.15.114:0/481072462 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f3db8052360 con 0x7f3dbc0a9d70 2024-03-29T10:41:44.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.506+0000 7f3db2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:44.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.506+0000 7f3db2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:44.490 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:41:44.490 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:44.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.508+0000 7f3dc437b640 10 librados: watch_flush enter 2024-03-29T10:41:44.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 10 librados: watch_flush exit 2024-03-29T10:41:44.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-29T10:41:44.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 10 client.4303.objecter successfully canceled tick 2024-03-29T10:41:44.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 1 -- 172.21.15.114:0/481072462 >> v1:172.21.15.114:6816/775044307 conn(0x7f3da006de50 legacy=0x7f3da0070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:44.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 10 monclient: shutdown 2024-03-29T10:41:44.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:44.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 1 -- 172.21.15.114:0/481072462 >> v1:172.21.15.176:6789/0 conn(0x7f3dbc0a9d70 legacy=0x7f3dbc1d8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:44.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 1 -- 172.21.15.114:0/481072462 shutdown_connections 2024-03-29T10:41:44.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 1 -- 172.21.15.114:0/481072462 >> 172.21.15.114:0/481072462 conn(0x7f3dbc066a00 msgr2=0x7f3dbc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:44.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 1 -- 172.21.15.114:0/481072462 shutdown_connections 2024-03-29T10:41:44.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 1 -- 172.21.15.114:0/481072462 wait complete. 2024-03-29T10:41:44.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:44.509+0000 7f3dc437b640 1 librados: shutdown 2024-03-29T10:41:44.506 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:51.313 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:51.462 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.478+0000 7f858a2e2640 10 monclient: get_monmap_and_config 2024-03-29T10:41:51.462 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.478+0000 7f858a2e2640 10 monclient: build_initial_monmap 2024-03-29T10:41:51.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.478+0000 7f858a2e2640 10 monclient: monmap: 2024-03-29T10:41:51.463 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:51.463 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:51.463 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:41:51.479887+0000 2024-03-29T10:41:51.463 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:41:51.479887+0000 2024-03-29T10:41:51.463 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:51.463 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:51.463 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:51.463 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:41:51.463 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:41:51.463 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:51.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.479+0000 7f858a2e2640 1 Processor -- start 2024-03-29T10:41:51.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.479+0000 7f858a2e2640 1 -- start start 2024-03-29T10:41:51.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.479+0000 7f858a2e2640 10 monclient: init 2024-03-29T10:41:51.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.480+0000 7f858a2e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:51.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.480+0000 7f858a2e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:51.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.480+0000 7f858a2e2640 10 monclient(hunting): picked mon.noname-b con 0x7f85840b0770 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:51.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.480+0000 7f858a2e2640 10 monclient(hunting): picked mon.noname-a con 0x7f858414a700 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:51.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.480+0000 7f858a2e2640 10 monclient(hunting): picked mon.noname-c con 0x7f85840ac5a0 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:51.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.480+0000 7f858a2e2640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85840aba70 con 0x7f858414a700 2024-03-29T10:41:51.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.480+0000 7f858a2e2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85840aecb0 con 0x7f85840ac5a0 2024-03-29T10:41:51.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.480+0000 7f858a2e2640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85840af150 con 0x7f85840b0770 2024-03-29T10:41:51.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.480+0000 7f858a2e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:51.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.480+0000 7f858a2e2640 10 monclient(hunting): authenticate will time out at 976.721547s 2024-03-29T10:41:51.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.481+0000 7f8583fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f85840b0770 0x7f8584149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:58650/0 (socket says 172.21.15.114:58650) 2024-03-29T10:41:51.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.481+0000 7f8583fff640 1 -- 172.21.15.114:0/874933320 learned_addr learned my addr 172.21.15.114:0/874933320 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:51.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.482+0000 7f8582ffd640 1 -- 172.21.15.114:0/874933320 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495448547 0 0) 0x7f85840af150 con 0x7f85840b0770 2024-03-29T10:41:51.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.482+0000 7f8582ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:51.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.482+0000 7f8582ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:51.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.482+0000 7f8582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:51.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.482+0000 7f8582ffd640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:41:51.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.482+0000 7f8582ffd640 1 -- 172.21.15.114:0/874933320 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f856c003520 con 0x7f85840b0770 2024-03-29T10:41:51.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 1 -- 172.21.15.114:0/874933320 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963055442 0 0) 0x7f85840aecb0 con 0x7f85840ac5a0 2024-03-29T10:41:51.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:51.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:51.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:51.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:41:51.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 1 -- 172.21.15.114:0/874933320 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85840af150 con 0x7f85840ac5a0 2024-03-29T10:41:51.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 1 -- 172.21.15.114:0/874933320 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1651546087 0 0) 0x7f856c003520 con 0x7f85840b0770 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 1 -- 172.21.15.114:0/874933320 >> v1:172.21.15.176:6789/0 conn(0x7f85840ac5a0 legacy=0x7f8584151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 1 -- 172.21.15.114:0/874933320 >> v1:172.21.15.114:6789/0 conn(0x7f858414a700 legacy=0x7f85840abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 1 monclient: found mon.noname-b 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 1 -- 172.21.15.114:0/874933320 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85840ac9d0 con 0x7f85840b0770 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 1 -- 172.21.15.114:0/874933320 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8578001610 con 0x7f85840b0770 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.484+0000 7f8582ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.485+0000 7f8582ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:51.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.485+0000 7f8582ffd640 10 monclient: dump: 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.485+0000 7f858a2e2640 5 monclient: authenticate success, global_id 4331 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.485+0000 7f858a2e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:51.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.485+0000 7f8582ffd640 1 -- 172.21.15.114:0/874933320 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8578001770 con 0x7f85840b0770 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.485+0000 7f8582ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.485+0000 7f8582ffd640 1 -- 172.21.15.114:0/874933320 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8578001bc0 con 0x7f85840b0770 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.485+0000 7f8582ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.485+0000 7f8582ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.485+0000 7f8582ffd640 10 monclient: dump: 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:51.475 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.486+0000 7f858a2e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.486+0000 7f858a2e2640 4 set_mon_vals no callback set 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.486+0000 7f858a2e2640 10 monclient: shutdown 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.486+0000 7f858a2e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.486+0000 7f858a2e2640 1 -- 172.21.15.114:0/874933320 >> v1:172.21.15.114:6790/0 conn(0x7f85840b0770 legacy=0x7f8584149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.487+0000 7f858a2e2640 1 -- 172.21.15.114:0/874933320 shutdown_connections 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.487+0000 7f858a2e2640 1 -- 172.21.15.114:0/874933320 >> 172.21.15.114:0/874933320 conn(0x7f858406b660 msgr2=0x7f858406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.487+0000 7f858a2e2640 1 -- 172.21.15.114:0/874933320 shutdown_connections 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.487+0000 7f858a2e2640 1 -- 172.21.15.114:0/874933320 wait complete. 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.487+0000 7f858a2e2640 10 monclient: build_initial_monmap 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.487+0000 7f858a2e2640 10 monclient: monmap: 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:51.476 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:41:51.477 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:41:51.477 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:51.477 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:51.477 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:51.477 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:41:51.477 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:41:51.477 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:51.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.488+0000 7f858a2e2640 1 librados: starting msgr at 2024-03-29T10:41:51.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.488+0000 7f858a2e2640 1 librados: starting objecter 2024-03-29T10:41:51.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.488+0000 7f858a2e2640 1 Processor -- start 2024-03-29T10:41:51.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.488+0000 7f858a2e2640 1 -- start start 2024-03-29T10:41:51.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.488+0000 7f858a2e2640 1 librados: setting wanted keys 2024-03-29T10:41:51.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.488+0000 7f858a2e2640 1 librados: calling monclient init 2024-03-29T10:41:51.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.488+0000 7f858a2e2640 10 monclient: init 2024-03-29T10:41:51.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.489+0000 7f858a2e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:51.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.489+0000 7f858a2e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:51.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.489+0000 7f858a2e2640 10 monclient(hunting): picked mon.noname-c con 0x7f85840ac5a0 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:51.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.489+0000 7f858a2e2640 10 monclient(hunting): picked mon.noname-a con 0x7f85840b0770 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:51.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.489+0000 7f858a2e2640 10 monclient(hunting): picked mon.noname-b con 0x7f858414a700 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:51.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.489+0000 7f858a2e2640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8584147a30 con 0x7f85840b0770 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.489+0000 7f858a2e2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85841dbe70 con 0x7f858414a700 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.489+0000 7f858a2e2640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85841dba50 con 0x7f85840ac5a0 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.489+0000 7f858a2e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.489+0000 7f858a2e2640 10 monclient(hunting): authenticate will time out at 976.730690s 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.493+0000 7f8583fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f85840ac5a0 0x7f85841db340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:59018/0 (socket says 172.21.15.114:59018) 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.493+0000 7f8583fff640 1 -- 172.21.15.114:0/1340326537 learned_addr learned my addr 172.21.15.114:0/1340326537 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 10 client.?.objecter ms_handle_connect 0x7f85840ac5a0 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 10 client.?.objecter ms_handle_connect 0x7f858414a700 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 10 client.?.objecter ms_handle_connect 0x7f85840b0770 2024-03-29T10:41:51.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387739568 0 0) 0x7f85841dba50 con 0x7f85840ac5a0 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8570004eb0 con 0x7f85840ac5a0 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.495+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265970380 0 0) 0x7f8584147a30 con 0x7f85840b0770 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): my global_id is 4329 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85841dba50 con 0x7f85840b0770 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896792642 0 0) 0x7f85841dbe70 con 0x7f858414a700 2024-03-29T10:41:51.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8584147a30 con 0x7f858414a700 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1495533890 0 0) 0x7f8570004eb0 con 0x7f85840ac5a0 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85841dbe70 con 0x7f85840ac5a0 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8578001b40 con 0x7f85840ac5a0 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): dump: 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:51.488 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1684277240 0 0) 0x7f85841dba50 con 0x7f85840b0770 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8570004eb0 con 0x7f85840b0770 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2113819200 0 0) 0x7f85841dbe70 con 0x7f85840ac5a0 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 >> v1:172.21.15.176:6789/0 conn(0x7f858414a700 legacy=0x7f85841e5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 >> v1:172.21.15.114:6789/0 conn(0x7f85840b0770 legacy=0x7f85841de840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 monclient: found mon.c 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:51.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8584070650 con 0x7f85840ac5a0 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient: _finish_auth 0 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f8580ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f858a2e2640 5 monclient: authenticate success, global_id 4334 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f858a2e2640 10 monclient: _renew_subs 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f858a2e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.496+0000 7f858a2e2640 1 -- 172.21.15.114:0/1340326537 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8584070840 con 0x7f85840ac5a0 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.497+0000 7f858a2e2640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.497+0000 7f858a2e2640 10 monclient: _renew_subs 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.497+0000 7f858a2e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.497+0000 7f858a2e2640 1 -- 172.21.15.114:0/1340326537 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85841df130 con 0x7f85840ac5a0 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.497+0000 7f858a2e2640 1 librados: init done 2024-03-29T10:41:51.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.497+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8578004a90 con 0x7f85840ac5a0 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.497+0000 7f8580ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.497+0000 7f85827fc640 4 set_mon_vals no callback set 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.499+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8578004da0 con 0x7f85840ac5a0 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.499+0000 7f8580ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.499+0000 7f8580ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.499+0000 7f8580ff9640 10 monclient: dump: 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:51.491 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.499+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f857801ad70 con 0x7f85840ac5a0 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.500+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f8578084bd0 con 0x7f85840ac5a0 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.500+0000 7f8580ff9640 10 client.4334.objecter ms_dispatch 0x7f8584000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.500+0000 7f8580ff9640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.500+0000 7f8580ff9640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.500+0000 7f8580ff9640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.500+0000 7f858a2e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.501+0000 7f85837fe640 10 monclient: get_auth_request con 0x7f857006dda0 auth_method 0 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.501+0000 7f858a2e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.501+0000 7f858a2e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.501+0000 7f858a2e2640 1 -- 172.21.15.114:0/1340326537 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8564005080 con 0x7f85840ac5a0 2024-03-29T10:41:51.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.501+0000 7f8580ff9640 10 client.4334.objecter ms_handle_connect 0x7f857006dda0 2024-03-29T10:41:51.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.509+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8578050530 con 0x7f85840ac5a0 2024-03-29T10:41:51.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.509+0000 7f8580ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:51.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.509+0000 7f8580ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:51.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.825+0000 7f858a2e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:51.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.825+0000 7f858a2e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:51.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.825+0000 7f858a2e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:51.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.825+0000 7f858a2e2640 1 -- 172.21.15.114:0/1340326537 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8564005330 con 0x7f85840ac5a0 2024-03-29T10:41:51.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.826+0000 7f8580ff9640 1 -- 172.21.15.114:0/1340326537 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f8578052460 con 0x7f85840ac5a0 2024-03-29T10:41:51.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.826+0000 7f8580ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:51.810 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:41:51.810 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:51.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.826+0000 7f8580ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:51.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.829+0000 7f858a2e2640 10 librados: watch_flush enter 2024-03-29T10:41:51.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.830+0000 7f858a2e2640 10 librados: watch_flush exit 2024-03-29T10:41:51.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.830+0000 7f858a2e2640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-29T10:41:51.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.830+0000 7f858a2e2640 10 client.4334.objecter successfully canceled tick 2024-03-29T10:41:51.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.830+0000 7f858a2e2640 1 -- 172.21.15.114:0/1340326537 >> v1:172.21.15.114:6816/775044307 conn(0x7f857006dda0 legacy=0x7f8570070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:51.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.830+0000 7f858a2e2640 10 monclient: shutdown 2024-03-29T10:41:51.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.830+0000 7f858a2e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:51.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.830+0000 7f858a2e2640 1 -- 172.21.15.114:0/1340326537 >> v1:172.21.15.114:6790/0 conn(0x7f85840ac5a0 legacy=0x7f85841db340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:51.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.830+0000 7f858a2e2640 1 -- 172.21.15.114:0/1340326537 shutdown_connections 2024-03-29T10:41:51.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.830+0000 7f858a2e2640 1 -- 172.21.15.114:0/1340326537 >> 172.21.15.114:0/1340326537 conn(0x7f858406b660 msgr2=0x7f85840b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:51.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.830+0000 7f858a2e2640 1 -- 172.21.15.114:0/1340326537 shutdown_connections 2024-03-29T10:41:51.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.830+0000 7f858a2e2640 1 -- 172.21.15.114:0/1340326537 wait complete. 2024-03-29T10:41:51.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:51.831+0000 7f858a2e2640 1 librados: shutdown 2024-03-29T10:41:51.829 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:58.636 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:58.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.806+0000 7fc5deb89640 10 monclient: get_monmap_and_config 2024-03-29T10:41:58.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.806+0000 7fc5deb89640 10 monclient: build_initial_monmap 2024-03-29T10:41:58.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.806+0000 7fc5deb89640 10 monclient: monmap: 2024-03-29T10:41:58.790 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:58.790 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:58.790 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:41:58.806586+0000 2024-03-29T10:41:58.790 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:41:58.806586+0000 2024-03-29T10:41:58.790 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:58.790 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:58.790 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:58.790 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:41:58.790 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:41:58.790 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:58.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.808+0000 7fc5deb89640 1 Processor -- start 2024-03-29T10:41:58.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.809+0000 7fc5deb89640 1 -- start start 2024-03-29T10:41:58.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.809+0000 7fc5deb89640 10 monclient: init 2024-03-29T10:41:58.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5deb89640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:58.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5deb89640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:58.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5deb89640 10 monclient(hunting): picked mon.noname-a con 0x7fc5d80a9930 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:58.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5deb89640 10 monclient(hunting): picked mon.noname-c con 0x7fc5d8149dc0 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:58.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5deb89640 10 monclient(hunting): picked mon.noname-b con 0x7fc5d814ff40 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:58.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5deb89640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d80aae60 con 0x7fc5d80a9930 2024-03-29T10:41:58.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5deb89640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d80b4310 con 0x7fc5d8149dc0 2024-03-29T10:41:58.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5deb89640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d80afdd0 con 0x7fc5d814ff40 2024-03-29T10:41:58.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5deb89640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:58.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5deb89640 10 monclient(hunting): authenticate will time out at 984.050650s 2024-03-29T10:41:58.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5dd126640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fc5d814ff40 0x7fc5d8152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:47482/0 (socket says 172.21.15.114:47482) 2024-03-29T10:41:58.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.810+0000 7fc5dd126640 1 -- 172.21.15.114:0/3942037322 learned_addr learned my addr 172.21.15.114:0/3942037322 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:58.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.811+0000 7fc5d77fe640 1 -- 172.21.15.114:0/3942037322 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478376238 0 0) 0x7fc5d80afdd0 con 0x7fc5d814ff40 2024-03-29T10:41:58.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.811+0000 7fc5d77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:58.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.811+0000 7fc5d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:58.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.811+0000 7fc5d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:58.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.811+0000 7fc5d77fe640 10 monclient(hunting): my global_id is 4337 2024-03-29T10:41:58.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.811+0000 7fc5d77fe640 1 -- 172.21.15.114:0/3942037322 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c0003520 con 0x7fc5d814ff40 2024-03-29T10:41:58.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.812+0000 7fc5d77fe640 1 -- 172.21.15.114:0/3942037322 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3180712790 0 0) 0x7fc5c0003520 con 0x7fc5d814ff40 2024-03-29T10:41:58.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.812+0000 7fc5d77fe640 1 -- 172.21.15.114:0/3942037322 >> v1:172.21.15.176:6789/0 conn(0x7fc5d8149dc0 legacy=0x7fc5d814a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:58.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 1 -- 172.21.15.114:0/3942037322 >> v1:172.21.15.114:6789/0 conn(0x7fc5d80a9930 legacy=0x7fc5d8143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:58.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:41:58.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 1 monclient: found mon.noname-b 2024-03-29T10:41:58.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:58.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 1 -- 172.21.15.114:0/3942037322 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d814a950 con 0x7fc5d814ff40 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 10 monclient: _finish_auth 0 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5deb89640 5 monclient: authenticate success, global_id 4337 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5deb89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 1 -- 172.21.15.114:0/3942037322 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc5d0001a10 con 0x7fc5d814ff40 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 10 monclient: dump: 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:58.801 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5deb89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 1 -- 172.21.15.114:0/3942037322 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5d0001ca0 con 0x7fc5d814ff40 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5deb89640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.815+0000 7fc5deb89640 4 set_mon_vals no callback set 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.816+0000 7fc5deb89640 10 monclient: shutdown 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.816+0000 7fc5deb89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:58.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.816+0000 7fc5deb89640 1 -- 172.21.15.114:0/3942037322 >> v1:172.21.15.114:6790/0 conn(0x7fc5d814ff40 legacy=0x7fc5d8152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:58.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.816+0000 7fc5deb89640 1 -- 172.21.15.114:0/3942037322 shutdown_connections 2024-03-29T10:41:58.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.816+0000 7fc5deb89640 1 -- 172.21.15.114:0/3942037322 >> 172.21.15.114:0/3942037322 conn(0x7fc5d80b35e0 msgr2=0x7fc5d80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:58.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.817+0000 7fc5deb89640 1 -- 172.21.15.114:0/3942037322 shutdown_connections 2024-03-29T10:41:58.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.817+0000 7fc5deb89640 1 -- 172.21.15.114:0/3942037322 wait complete. 2024-03-29T10:41:58.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.817+0000 7fc5deb89640 10 monclient: build_initial_monmap 2024-03-29T10:41:58.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.817+0000 7fc5deb89640 10 monclient: monmap: 2024-03-29T10:41:58.804 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:41:58.804 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:58.804 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:41:58.804 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:41:58.804 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:58.804 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.818+0000 7fc5deb89640 1 librados: starting msgr at 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.818+0000 7fc5deb89640 1 librados: starting objecter 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.818+0000 7fc5deb89640 1 Processor -- start 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.818+0000 7fc5deb89640 1 -- start start 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.818+0000 7fc5deb89640 1 librados: setting wanted keys 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.818+0000 7fc5deb89640 1 librados: calling monclient init 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.818+0000 7fc5deb89640 10 monclient: init 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.819+0000 7fc5deb89640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.819+0000 7fc5deb89640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.819+0000 7fc5deb89640 10 monclient(hunting): picked mon.noname-b con 0x7fc5d80a9930 addr v1:172.21.15.176:6789/0 2024-03-29T10:41:58.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.819+0000 7fc5deb89640 10 monclient(hunting): picked mon.noname-c con 0x7fc5d8149dc0 addr v1:172.21.15.114:6790/0 2024-03-29T10:41:58.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.819+0000 7fc5deb89640 10 monclient(hunting): picked mon.noname-a con 0x7fc5d814ff40 addr v1:172.21.15.114:6789/0 2024-03-29T10:41:58.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.819+0000 7fc5deb89640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d81489c0 con 0x7fc5d814ff40 2024-03-29T10:41:58.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.819+0000 7fc5deb89640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d8145ff0 con 0x7fc5d80a9930 2024-03-29T10:41:58.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.819+0000 7fc5deb89640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d8149a60 con 0x7fc5d8149dc0 2024-03-29T10:41:58.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.819+0000 7fc5deb89640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:58.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.819+0000 7fc5deb89640 10 monclient(hunting): authenticate will time out at 984.059538s 2024-03-29T10:41:58.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.823+0000 7fc5d7fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fc5d8149dc0 0x7fc5d8146510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:48108/0 (socket says 172.21.15.114:48108) 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.823+0000 7fc5d7fff640 1 -- 172.21.15.114:0/3661396891 learned_addr learned my addr 172.21.15.114:0/3661396891 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 client.?.objecter ms_handle_connect 0x7fc5d814ff40 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 client.?.objecter ms_handle_connect 0x7fc5d8149dc0 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324292272 0 0) 0x7fc5d81489c0 con 0x7fc5d814ff40 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c4004eb0 con 0x7fc5d814ff40 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421721559 0 0) 0x7fc5d8149a60 con 0x7fc5d8149dc0 2024-03-29T10:41:58.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.824+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d81489c0 con 0x7fc5d8149dc0 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3681656938 0 0) 0x7fc5d81489c0 con 0x7fc5d8149dc0 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5c4005490 con 0x7fc5d8149dc0 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc5cc0017f0 con 0x7fc5d8149dc0 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 monclient(hunting): dump: 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:58.810 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 client.?.objecter ms_handle_connect 0x7fc5d80a9930 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1951764939 0 0) 0x7fc5c4004eb0 con 0x7fc5d814ff40 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5c4005b40 con 0x7fc5d814ff40 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511948046 0 0) 0x7fc5d8145ff0 con 0x7fc5d80a9930 2024-03-29T10:41:58.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:58.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:58.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:58.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:41:58.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c4004eb0 con 0x7fc5d80a9930 2024-03-29T10:41:58.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc5d0004d00 con 0x7fc5d814ff40 2024-03-29T10:41:58.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:58.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.825+0000 7fc5d57fa640 10 monclient(hunting): dump: 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1495493920 0 0) 0x7fc5c4005490 con 0x7fc5d8149dc0 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 >> v1:172.21.15.176:6789/0 conn(0x7fc5d80a9930 legacy=0x7fc5d8149350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:58.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 >> v1:172.21.15.114:6789/0 conn(0x7fc5d814ff40 legacy=0x7fc5d8146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5d57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5d57fa640 1 monclient: found mon.c 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5d57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8148c00 con 0x7fc5d8149dc0 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5d57fa640 10 monclient: _finish_auth 0 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5d57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5deb89640 5 monclient: authenticate success, global_id 4340 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5deb89640 10 monclient: _renew_subs 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5deb89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5deb89640 1 -- 172.21.15.114:0/3661396891 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d8148eb0 con 0x7fc5d8149dc0 2024-03-29T10:41:58.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5deb89640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5deb89640 10 monclient: _renew_subs 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5deb89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5deb89640 1 -- 172.21.15.114:0/3661396891 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d81e71d0 con 0x7fc5d8149dc0 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.826+0000 7fc5deb89640 1 librados: init done 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.827+0000 7fc5deb89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.827+0000 7fc5deb89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.827+0000 7fc5deb89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.827+0000 7fc5deb89640 1 -- 172.21.15.114:0/3661396891 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d8146230 con 0x7fc5d8149dc0 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.829+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5cc001d40 con 0x7fc5d8149dc0 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.829+0000 7fc5d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.829+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc5cc004900 con 0x7fc5d8149dc0 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.829+0000 7fc5d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.829+0000 7fc5d57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:58.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.829+0000 7fc5d57fa640 10 monclient: dump: 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.829+0000 7fc5d67fc640 4 set_mon_vals no callback set 2024-03-29T10:41:58.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.829+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fc5cc01a8d0 con 0x7fc5d8149dc0 2024-03-29T10:41:58.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.830+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fc5cc084830 con 0x7fc5d8149dc0 2024-03-29T10:41:58.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.830+0000 7fc5d57fa640 10 client.4340.objecter ms_dispatch 0x7fc5d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:58.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.830+0000 7fc5d57fa640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:58.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.830+0000 7fc5d57fa640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:58.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.830+0000 7fc5d57fa640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T10:41:58.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.830+0000 7fc5dc925640 10 monclient: get_auth_request con 0x7fc5c406e390 auth_method 0 2024-03-29T10:41:58.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.831+0000 7fc5d57fa640 10 client.4340.objecter ms_handle_connect 0x7fc5c406e390 2024-03-29T10:41:58.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.837+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5cc050190 con 0x7fc5d8149dc0 2024-03-29T10:41:58.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.837+0000 7fc5d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:58.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:58.837+0000 7fc5d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:59.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.181+0000 7fc5deb89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:59.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.181+0000 7fc5deb89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:59.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.181+0000 7fc5deb89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:41:59.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.181+0000 7fc5deb89640 1 -- 172.21.15.114:0/3661396891 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5d8075f10 con 0x7fc5d8149dc0 2024-03-29T10:41:59.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.183+0000 7fc5d57fa640 1 -- 172.21.15.114:0/3661396891 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fc5cc0520c0 con 0x7fc5d8149dc0 2024-03-29T10:41:59.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.183+0000 7fc5d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:59.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.183+0000 7fc5d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:59.166 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:41:59.167 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:59.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.186+0000 7fc5deb89640 10 librados: watch_flush enter 2024-03-29T10:41:59.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.186+0000 7fc5deb89640 10 librados: watch_flush exit 2024-03-29T10:41:59.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.186+0000 7fc5deb89640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T10:41:59.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.186+0000 7fc5deb89640 10 client.4340.objecter successfully canceled tick 2024-03-29T10:41:59.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.187+0000 7fc5deb89640 1 -- 172.21.15.114:0/3661396891 >> v1:172.21.15.114:6816/775044307 conn(0x7fc5c406e390 legacy=0x7fc5c4070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:59.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.187+0000 7fc5deb89640 10 monclient: shutdown 2024-03-29T10:41:59.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.187+0000 7fc5deb89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:59.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.187+0000 7fc5deb89640 1 -- 172.21.15.114:0/3661396891 >> v1:172.21.15.114:6790/0 conn(0x7fc5d8149dc0 legacy=0x7fc5d8146510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:59.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.187+0000 7fc5deb89640 1 -- 172.21.15.114:0/3661396891 shutdown_connections 2024-03-29T10:41:59.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.187+0000 7fc5deb89640 1 -- 172.21.15.114:0/3661396891 >> 172.21.15.114:0/3661396891 conn(0x7fc5d80b35e0 msgr2=0x7fc5d80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:59.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.187+0000 7fc5deb89640 1 -- 172.21.15.114:0/3661396891 shutdown_connections 2024-03-29T10:41:59.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.187+0000 7fc5deb89640 1 -- 172.21.15.114:0/3661396891 wait complete. 2024-03-29T10:41:59.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:41:59.187+0000 7fc5deb89640 1 librados: shutdown 2024-03-29T10:41:59.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:05.995 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:06.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.154+0000 7f6e611f3640 10 monclient: get_monmap_and_config 2024-03-29T10:42:06.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.154+0000 7f6e611f3640 10 monclient: build_initial_monmap 2024-03-29T10:42:06.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.154+0000 7f6e611f3640 10 monclient: monmap: 2024-03-29T10:42:06.138 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:06.138 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:06.138 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:42:06.154800+0000 2024-03-29T10:42:06.138 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:42:06.154800+0000 2024-03-29T10:42:06.139 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:06.139 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:06.139 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:06.139 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:42:06.139 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:42:06.139 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:06.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.155+0000 7f6e611f3640 1 Processor -- start 2024-03-29T10:42:06.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.156+0000 7f6e611f3640 1 -- start start 2024-03-29T10:42:06.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.156+0000 7f6e611f3640 10 monclient: init 2024-03-29T10:42:06.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.156+0000 7f6e611f3640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:06.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.156+0000 7f6e611f3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:06.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.156+0000 7f6e611f3640 10 monclient(hunting): picked mon.noname-a con 0x7f6e5c0aaee0 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:06.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.156+0000 7f6e611f3640 10 monclient(hunting): picked mon.noname-c con 0x7f6e5c0aa500 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:06.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.157+0000 7f6e611f3640 10 monclient(hunting): picked mon.noname-b con 0x7f6e5c149800 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:06.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.157+0000 7f6e611f3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c0aded0 con 0x7f6e5c0aaee0 2024-03-29T10:42:06.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.157+0000 7f6e611f3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c0ad7e0 con 0x7f6e5c0aa500 2024-03-29T10:42:06.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.157+0000 7f6e611f3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c0ac350 con 0x7f6e5c149800 2024-03-29T10:42:06.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.157+0000 7f6e611f3640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:06.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.157+0000 7f6e611f3640 10 monclient(hunting): authenticate will time out at 991.397050s 2024-03-29T10:42:06.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.160+0000 7f6e609f2640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f6e5c149800 0x7f6e5c151ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33304/0 (socket says 172.21.15.114:33304) 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.160+0000 7f6e609f2640 1 -- 172.21.15.114:0/3537929075 learned_addr learned my addr 172.21.15.114:0/3537929075 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.160+0000 7f6e5affd640 1 -- 172.21.15.114:0/3537929075 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515873438 0 0) 0x7f6e5c0ac350 con 0x7f6e5c149800 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 1 -- 172.21.15.114:0/3537929075 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e48003520 con 0x7f6e5c149800 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 1 -- 172.21.15.114:0/3537929075 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562285974 0 0) 0x7f6e5c0aded0 con 0x7f6e5c0aaee0 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:06.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 1 -- 172.21.15.114:0/3537929075 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e5c0ac350 con 0x7f6e5c0aaee0 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 1 -- 172.21.15.114:0/3537929075 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 372192664 0 0) 0x7f6e48003520 con 0x7f6e5c149800 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 1 -- 172.21.15.114:0/3537929075 >> v1:172.21.15.176:6789/0 conn(0x7f6e5c0aa500 legacy=0x7f6e5c0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 1 -- 172.21.15.114:0/3537929075 >> v1:172.21.15.114:6789/0 conn(0x7f6e5c0aaee0 legacy=0x7f6e5c14a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 1 monclient: found mon.noname-b 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 1 -- 172.21.15.114:0/3537929075 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c0ac630 con 0x7f6e5c149800 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient: _finish_auth 0 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 1 -- 172.21.15.114:0/3537929075 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6e54001610 con 0x7f6e5c149800 2024-03-29T10:42:06.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient: dump: 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:06.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e611f3640 5 monclient: authenticate success, global_id 4343 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e611f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 1 -- 172.21.15.114:0/3537929075 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e54001770 con 0x7f6e5c149800 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e611f3640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.161+0000 7f6e611f3640 4 set_mon_vals no callback set 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.162+0000 7f6e5affd640 1 -- 172.21.15.114:0/3537929075 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6e54001bc0 con 0x7f6e5c149800 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.162+0000 7f6e5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.162+0000 7f6e5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.162+0000 7f6e5affd640 10 monclient: dump: 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:06.151 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.162+0000 7f6e611f3640 10 monclient: shutdown 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.162+0000 7f6e611f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.162+0000 7f6e611f3640 1 -- 172.21.15.114:0/3537929075 >> v1:172.21.15.114:6790/0 conn(0x7f6e5c149800 legacy=0x7f6e5c151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.162+0000 7f6e611f3640 1 -- 172.21.15.114:0/3537929075 shutdown_connections 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.162+0000 7f6e611f3640 1 -- 172.21.15.114:0/3537929075 >> 172.21.15.114:0/3537929075 conn(0x7f6e5c0b3a30 msgr2=0x7f6e5c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.163+0000 7f6e611f3640 1 -- 172.21.15.114:0/3537929075 shutdown_connections 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.163+0000 7f6e611f3640 1 -- 172.21.15.114:0/3537929075 wait complete. 2024-03-29T10:42:06.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.164+0000 7f6e611f3640 10 monclient: build_initial_monmap 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.164+0000 7f6e611f3640 10 monclient: monmap: 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.164+0000 7f6e611f3640 1 librados: starting msgr at 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.164+0000 7f6e611f3640 1 librados: starting objecter 2024-03-29T10:42:06.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.164+0000 7f6e611f3640 1 Processor -- start 2024-03-29T10:42:06.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.165+0000 7f6e611f3640 1 -- start start 2024-03-29T10:42:06.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.165+0000 7f6e611f3640 1 librados: setting wanted keys 2024-03-29T10:42:06.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.165+0000 7f6e611f3640 1 librados: calling monclient init 2024-03-29T10:42:06.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.165+0000 7f6e611f3640 10 monclient: init 2024-03-29T10:42:06.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.166+0000 7f6e611f3640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:06.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.166+0000 7f6e611f3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.166+0000 7f6e611f3640 10 monclient(hunting): picked mon.noname-b con 0x7f6e5c0aa500 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.166+0000 7f6e611f3640 10 monclient(hunting): picked mon.noname-a con 0x7f6e5c0aaee0 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.166+0000 7f6e611f3640 10 monclient(hunting): picked mon.noname-c con 0x7f6e5c149800 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.166+0000 7f6e611f3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c139d70 con 0x7f6e5c0aaee0 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.166+0000 7f6e611f3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c140f80 con 0x7f6e5c0aa500 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.166+0000 7f6e611f3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c13e960 con 0x7f6e5c149800 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.166+0000 7f6e611f3640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.166+0000 7f6e611f3640 10 monclient(hunting): authenticate will time out at 991.406829s 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.168+0000 7f6e609f2640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f6e5c149800 0x7f6e5c1e5af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:36242/0 (socket says 172.21.15.114:36242) 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.168+0000 7f6e609f2640 1 -- 172.21.15.114:0/659046272 learned_addr learned my addr 172.21.15.114:0/659046272 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 client.?.objecter ms_handle_connect 0x7f6e5c149800 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:06.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413727451 0 0) 0x7f6e5c13e960 con 0x7f6e5c149800 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e3c004eb0 con 0x7f6e5c149800 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 client.?.objecter ms_handle_connect 0x7f6e5c0aa500 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360897631 0 0) 0x7f6e5c140f80 con 0x7f6e5c0aa500 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:42:06.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e5c13e960 con 0x7f6e5c0aa500 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2892571255 0 0) 0x7f6e5c13e960 con 0x7f6e5c0aa500 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.170+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e5c140f80 con 0x7f6e5c0aa500 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6e440017e0 con 0x7f6e5c0aa500 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 10 monclient(hunting): dump: 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:06.162 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 10 client.?.objecter ms_handle_connect 0x7f6e5c0aaee0 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431790660 0 0) 0x7f6e5c139d70 con 0x7f6e5c0aaee0 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.171+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e5c13e960 con 0x7f6e5c0aaee0 2024-03-29T10:42:06.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.172+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2149690367 0 0) 0x7f6e5c13e960 con 0x7f6e5c0aaee0 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.172+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e5c139d70 con 0x7f6e5c0aaee0 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.172+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6e4c001b50 con 0x7f6e5c0aaee0 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.172+0000 7f6e58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.172+0000 7f6e58ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.172+0000 7f6e58ff9640 10 monclient(hunting): dump: 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:06.164 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1437440601 0 0) 0x7f6e5c140f80 con 0x7f6e5c0aa500 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 >> v1:172.21.15.114:6790/0 conn(0x7f6e5c149800 legacy=0x7f6e5c1e5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 >> v1:172.21.15.114:6789/0 conn(0x7f6e5c0aaee0 legacy=0x7f6e5c145550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e58ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e58ff9640 1 monclient: found mon.b 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e58ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e58ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c13eaf0 con 0x7f6e5c0aa500 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e58ff9640 10 monclient: _finish_auth 0 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e58ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e611f3640 5 monclient: authenticate success, global_id 4315 2024-03-29T10:42:06.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e611f3640 10 monclient: _renew_subs 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e611f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e611f3640 1 -- 172.21.15.114:0/659046272 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c143890 con 0x7f6e5c0aa500 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e611f3640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e611f3640 10 monclient: _renew_subs 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e611f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e611f3640 1 -- 172.21.15.114:0/659046272 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c13f7f0 con 0x7f6e5c0aa500 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.173+0000 7f6e611f3640 1 librados: init done 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.174+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e44001940 con 0x7f6e5c0aa500 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.174+0000 7f6e58ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.174+0000 7f6e5a7fc640 4 set_mon_vals no callback set 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.174+0000 7f6e611f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.174+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6e440037b0 con 0x7f6e5c0aa500 2024-03-29T10:42:06.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.175+0000 7f6e611f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.175+0000 7f6e611f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.175+0000 7f6e611f3640 1 -- 172.21.15.114:0/659046272 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e40005080 con 0x7f6e5c0aa500 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.176+0000 7f6e58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.176+0000 7f6e58ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.176+0000 7f6e58ff9640 10 monclient: dump: 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:06.167 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:06.168 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:06.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.176+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f6e44003910 con 0x7f6e5c0aa500 2024-03-29T10:42:06.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.177+0000 7f6e5b7fe640 10 monclient: get_auth_request con 0x7f6e3c06dfb0 auth_method 0 2024-03-29T10:42:06.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.177+0000 7f6e58ff9640 10 client.4315.objecter ms_handle_connect 0x7f6e3c06dfb0 2024-03-29T10:42:06.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.177+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f6e440837b0 con 0x7f6e5c0aa500 2024-03-29T10:42:06.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.177+0000 7f6e58ff9640 10 client.4315.objecter ms_dispatch 0x7f6e5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:06.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.177+0000 7f6e58ff9640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:06.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.177+0000 7f6e58ff9640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:06.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.178+0000 7f6e58ff9640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-29T10:42:06.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.185+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e4404f090 con 0x7f6e5c0aa500 2024-03-29T10:42:06.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.185+0000 7f6e58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:06.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.185+0000 7f6e58ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:06.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.510+0000 7f6e611f3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:06.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.510+0000 7f6e611f3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:06.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.510+0000 7f6e611f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:06.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.510+0000 7f6e611f3640 1 -- 172.21.15.114:0/659046272 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e40005330 con 0x7f6e5c0aa500 2024-03-29T10:42:06.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.511+0000 7f6e58ff9640 1 -- 172.21.15.114:0/659046272 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f6e44050fc0 con 0x7f6e5c0aa500 2024-03-29T10:42:06.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.511+0000 7f6e58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:06.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.511+0000 7f6e58ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:06.495 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:42:06.495 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:06.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.515+0000 7f6e611f3640 10 librados: watch_flush enter 2024-03-29T10:42:06.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.515+0000 7f6e611f3640 10 librados: watch_flush exit 2024-03-29T10:42:06.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.515+0000 7f6e611f3640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-29T10:42:06.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.515+0000 7f6e611f3640 10 client.4315.objecter successfully canceled tick 2024-03-29T10:42:06.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.515+0000 7f6e611f3640 1 -- 172.21.15.114:0/659046272 >> v1:172.21.15.114:6816/775044307 conn(0x7f6e3c06dfb0 legacy=0x7f6e3c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:06.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.515+0000 7f6e611f3640 10 monclient: shutdown 2024-03-29T10:42:06.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.515+0000 7f6e611f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:06.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.515+0000 7f6e611f3640 1 -- 172.21.15.114:0/659046272 >> v1:172.21.15.176:6789/0 conn(0x7f6e5c0aa500 legacy=0x7f6e5c144e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:06.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.515+0000 7f6e611f3640 1 -- 172.21.15.114:0/659046272 shutdown_connections 2024-03-29T10:42:06.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.515+0000 7f6e611f3640 1 -- 172.21.15.114:0/659046272 >> 172.21.15.114:0/659046272 conn(0x7f6e5c0b3a30 msgr2=0x7f6e5c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:06.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.516+0000 7f6e611f3640 1 -- 172.21.15.114:0/659046272 shutdown_connections 2024-03-29T10:42:06.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.516+0000 7f6e611f3640 1 -- 172.21.15.114:0/659046272 wait complete. 2024-03-29T10:42:06.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:06.516+0000 7f6e611f3640 1 librados: shutdown 2024-03-29T10:42:06.516 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:13.323 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:13.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.489+0000 7fc168be8640 10 monclient: get_monmap_and_config 2024-03-29T10:42:13.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.489+0000 7fc168be8640 10 monclient: build_initial_monmap 2024-03-29T10:42:13.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.489+0000 7fc168be8640 10 monclient: monmap: 2024-03-29T10:42:13.474 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:13.474 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:13.474 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:42:13.490976+0000 2024-03-29T10:42:13.475 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:42:13.490976+0000 2024-03-29T10:42:13.475 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:13.475 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:13.475 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:13.475 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:42:13.475 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:42:13.475 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:13.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.491+0000 7fc168be8640 1 Processor -- start 2024-03-29T10:42:13.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.491+0000 7fc168be8640 1 -- start start 2024-03-29T10:42:13.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.491+0000 7fc168be8640 10 monclient: init 2024-03-29T10:42:13.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.492+0000 7fc168be8640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:13.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.492+0000 7fc168be8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:13.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.492+0000 7fc168be8640 10 monclient(hunting): picked mon.noname-a con 0x7fc1640a9ac0 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:13.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.492+0000 7fc168be8640 10 monclient(hunting): picked mon.noname-b con 0x7fc16413ee90 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:13.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.492+0000 7fc168be8640 10 monclient(hunting): picked mon.noname-c con 0x7fc1640af900 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:13.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.492+0000 7fc168be8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1640afd10 con 0x7fc1640a9ac0 2024-03-29T10:42:13.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.492+0000 7fc168be8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1640af6f0 con 0x7fc1640af900 2024-03-29T10:42:13.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.492+0000 7fc168be8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1640aac80 con 0x7fc16413ee90 2024-03-29T10:42:13.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.492+0000 7fc168be8640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:13.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.492+0000 7fc168be8640 10 monclient(hunting): authenticate will time out at 998.733304s 2024-03-29T10:42:13.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.495+0000 7fc16259c640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc1640a9ac0 0x7fc1640adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58017/0 (socket says 172.21.15.114:58017) 2024-03-29T10:42:13.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.495+0000 7fc16259c640 1 -- 172.21.15.114:0/3637187409 learned_addr learned my addr 172.21.15.114:0/3637187409 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:13.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 1 -- 172.21.15.114:0/3637187409 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042582784 0 0) 0x7fc1640afd10 con 0x7fc1640a9ac0 2024-03-29T10:42:13.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:13.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:13.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:13.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 10 monclient(hunting): my global_id is 4344 2024-03-29T10:42:13.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 1 -- 172.21.15.114:0/3637187409 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc14c003520 con 0x7fc1640a9ac0 2024-03-29T10:42:13.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 1 -- 172.21.15.114:0/3637187409 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310305193 0 0) 0x7fc1640af6f0 con 0x7fc1640af900 2024-03-29T10:42:13.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:13.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 10 monclient(hunting): my global_id is 4318 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 1 -- 172.21.15.114:0/3637187409 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1640afd10 con 0x7fc1640af900 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.499+0000 7fc16159a640 1 -- 172.21.15.114:0/3637187409 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095494032 0 0) 0x7fc14c003520 con 0x7fc1640a9ac0 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 1 -- 172.21.15.114:0/3637187409 >> v1:172.21.15.114:6790/0 conn(0x7fc16413ee90 legacy=0x7fc16413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 1 -- 172.21.15.114:0/3637187409 >> v1:172.21.15.176:6789/0 conn(0x7fc1640af900 legacy=0x7fc164149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 1 monclient: found mon.noname-a 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 1 -- 172.21.15.114:0/3637187409 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1640aaf60 con 0x7fc1640a9ac0 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 10 monclient: _finish_auth 0 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:13.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 1 -- 172.21.15.114:0/3637187409 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc154004c90 con 0x7fc1640a9ac0 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.500+0000 7fc16159a640 10 monclient: dump: 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:13.485 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:13.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.501+0000 7fc16159a640 1 -- 172.21.15.114:0/3637187409 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc154004df0 con 0x7fc1640a9ac0 2024-03-29T10:42:13.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.501+0000 7fc16159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:13.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.501+0000 7fc16159a640 1 -- 172.21.15.114:0/3637187409 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc154001410 con 0x7fc1640a9ac0 2024-03-29T10:42:13.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.501+0000 7fc16159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:13.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.501+0000 7fc16159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:13.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.501+0000 7fc16159a640 10 monclient: dump: 2024-03-29T10:42:13.487 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:13.487 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:13.487 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:13.487 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:13.487 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.501+0000 7fc168be8640 5 monclient: authenticate success, global_id 4344 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.501+0000 7fc168be8640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.501+0000 7fc168be8640 4 set_mon_vals no callback set 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.502+0000 7fc168be8640 10 monclient: shutdown 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.502+0000 7fc168be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.502+0000 7fc168be8640 1 -- 172.21.15.114:0/3637187409 >> v1:172.21.15.114:6789/0 conn(0x7fc1640a9ac0 legacy=0x7fc1640adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.502+0000 7fc168be8640 1 -- 172.21.15.114:0/3637187409 shutdown_connections 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.502+0000 7fc168be8640 1 -- 172.21.15.114:0/3637187409 >> 172.21.15.114:0/3637187409 conn(0x7fc1640b35e0 msgr2=0x7fc1640b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:13.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.503+0000 7fc168be8640 1 -- 172.21.15.114:0/3637187409 shutdown_connections 2024-03-29T10:42:13.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.503+0000 7fc168be8640 1 -- 172.21.15.114:0/3637187409 wait complete. 2024-03-29T10:42:13.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.504+0000 7fc168be8640 10 monclient: build_initial_monmap 2024-03-29T10:42:13.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.504+0000 7fc168be8640 10 monclient: monmap: 2024-03-29T10:42:13.489 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:13.489 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:13.489 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:42:13.489 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:42:13.489 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:13.490 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:13.490 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:13.490 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:42:13.490 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:42:13.490 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:13.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.504+0000 7fc168be8640 1 librados: starting msgr at 2024-03-29T10:42:13.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.504+0000 7fc168be8640 1 librados: starting objecter 2024-03-29T10:42:13.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.504+0000 7fc168be8640 1 Processor -- start 2024-03-29T10:42:13.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.510+0000 7fc168be8640 1 -- start start 2024-03-29T10:42:13.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.510+0000 7fc168be8640 1 librados: setting wanted keys 2024-03-29T10:42:13.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.510+0000 7fc168be8640 1 librados: calling monclient init 2024-03-29T10:42:13.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.510+0000 7fc168be8640 10 monclient: init 2024-03-29T10:42:13.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.510+0000 7fc168be8640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:13.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.510+0000 7fc168be8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:13.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.510+0000 7fc168be8640 10 monclient(hunting): picked mon.noname-b con 0x7fc1640a9ac0 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:13.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.510+0000 7fc168be8640 10 monclient(hunting): picked mon.noname-a con 0x7fc1640af900 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:13.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.510+0000 7fc168be8640 10 monclient(hunting): picked mon.noname-c con 0x7fc16413ee90 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:13.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.510+0000 7fc168be8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1641382a0 con 0x7fc1640af900 2024-03-29T10:42:13.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.510+0000 7fc168be8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16406fd90 con 0x7fc1640a9ac0 2024-03-29T10:42:13.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.511+0000 7fc168be8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16406eb60 con 0x7fc16413ee90 2024-03-29T10:42:13.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.511+0000 7fc168be8640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:13.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.511+0000 7fc168be8640 10 monclient(hunting): authenticate will time out at 998.752017s 2024-03-29T10:42:13.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.514+0000 7fc162d9d640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fc16413ee90 0x7fc164137b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:58739/0 (socket says 172.21.15.114:58739) 2024-03-29T10:42:13.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.514+0000 7fc162d9d640 1 -- 172.21.15.114:0/1132864553 learned_addr learned my addr 172.21.15.114:0/1132864553 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:13.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.517+0000 7fc152ffd640 10 client.?.objecter ms_handle_connect 0x7fc16413ee90 2024-03-29T10:42:13.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.517+0000 7fc152ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:13.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.518+0000 7fc152ffd640 10 client.?.objecter ms_handle_connect 0x7fc1640af900 2024-03-29T10:42:13.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.518+0000 7fc152ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:13.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.518+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636607904 0 0) 0x7fc16406eb60 con 0x7fc16413ee90 2024-03-29T10:42:13.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.518+0000 7fc152ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:13.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.518+0000 7fc152ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:13.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.518+0000 7fc152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:13.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.518+0000 7fc152ffd640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:42:13.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.518+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc144004eb0 con 0x7fc16413ee90 2024-03-29T10:42:13.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.518+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421531381 0 0) 0x7fc1641382a0 con 0x7fc1640af900 2024-03-29T10:42:13.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:13.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:13.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:13.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:42:13.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc16406eb60 con 0x7fc1640af900 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3938438390 0 0) 0x7fc16406eb60 con 0x7fc1640af900 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1641382a0 con 0x7fc1640af900 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc158001810 con 0x7fc1640af900 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:13.509 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.519+0000 7fc152ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 10 client.?.objecter ms_handle_connect 0x7fc1640a9ac0 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280811460 0 0) 0x7fc16406fd90 con 0x7fc1640a9ac0 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc16406eb60 con 0x7fc1640a9ac0 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1974210748 0 0) 0x7fc1641382a0 con 0x7fc1640af900 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 >> v1:172.21.15.114:6790/0 conn(0x7fc16413ee90 legacy=0x7fc164137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 >> v1:172.21.15.176:6789/0 conn(0x7fc1640a9ac0 legacy=0x7fc1641d3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.523+0000 7fc152ffd640 1 monclient: found mon.a 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16406ecf0 con 0x7fc1640af900 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc158001cc0 con 0x7fc1640af900 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc1580037b0 con 0x7fc1640af900 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc152ffd640 10 monclient: dump: 2024-03-29T10:42:13.511 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc168be8640 5 monclient: authenticate success, global_id 4347 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc168be8640 10 monclient: _renew_subs 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc168be8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc168be8640 1 -- 172.21.15.114:0/1132864553 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1641c7940 con 0x7fc1640af900 2024-03-29T10:42:13.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc168be8640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:13.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc168be8640 10 monclient: _renew_subs 2024-03-29T10:42:13.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc168be8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:13.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc168be8640 1 -- 172.21.15.114:0/1132864553 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1641d3940 con 0x7fc1640af900 2024-03-29T10:42:13.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.524+0000 7fc168be8640 1 librados: init done 2024-03-29T10:42:13.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.525+0000 7fc160d99640 4 set_mon_vals no callback set 2024-03-29T10:42:13.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.529+0000 7fc168be8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:13.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.530+0000 7fc168be8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:13.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.530+0000 7fc168be8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:13.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.530+0000 7fc168be8640 1 -- 172.21.15.114:0/1132864553 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc148005080 con 0x7fc1640af900 2024-03-29T10:42:13.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.530+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fc158019780 con 0x7fc1640af900 2024-03-29T10:42:13.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.531+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fc1580836f0 con 0x7fc1640af900 2024-03-29T10:42:13.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.531+0000 7fc152ffd640 10 client.4347.objecter ms_dispatch 0x7fc164000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:13.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.531+0000 7fc152ffd640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:13.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.531+0000 7fc152ffd640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:13.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.531+0000 7fc16259c640 10 monclient: get_auth_request con 0x7fc14406dc50 auth_method 0 2024-03-29T10:42:13.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.531+0000 7fc152ffd640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T10:42:13.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.531+0000 7fc152ffd640 10 client.4347.objecter ms_handle_connect 0x7fc14406dc50 2024-03-29T10:42:13.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.540+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc15804efd0 con 0x7fc1640af900 2024-03-29T10:42:13.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.540+0000 7fc152ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:13.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.540+0000 7fc152ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:13.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.860+0000 7fc168be8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:13.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.860+0000 7fc168be8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:13.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.860+0000 7fc168be8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:13.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.860+0000 7fc168be8640 1 -- 172.21.15.114:0/1132864553 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc148005330 con 0x7fc1640af900 2024-03-29T10:42:13.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.865+0000 7fc152ffd640 1 -- 172.21.15.114:0/1132864553 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fc158050f00 con 0x7fc1640af900 2024-03-29T10:42:13.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.865+0000 7fc152ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:13.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.865+0000 7fc152ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:13.851 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:42:13.851 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:13.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.870+0000 7fc168be8640 10 librados: watch_flush enter 2024-03-29T10:42:13.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.870+0000 7fc168be8640 10 librados: watch_flush exit 2024-03-29T10:42:13.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.870+0000 7fc168be8640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T10:42:13.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.870+0000 7fc168be8640 10 client.4347.objecter successfully canceled tick 2024-03-29T10:42:13.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.871+0000 7fc168be8640 1 -- 172.21.15.114:0/1132864553 >> v1:172.21.15.114:6816/775044307 conn(0x7fc14406dc50 legacy=0x7fc144070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.871+0000 7fc168be8640 10 monclient: shutdown 2024-03-29T10:42:13.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.871+0000 7fc168be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:13.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.871+0000 7fc168be8640 1 -- 172.21.15.114:0/1132864553 >> v1:172.21.15.114:6789/0 conn(0x7fc1640af900 legacy=0x7fc164137480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.871+0000 7fc168be8640 1 -- 172.21.15.114:0/1132864553 shutdown_connections 2024-03-29T10:42:13.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.871+0000 7fc168be8640 1 -- 172.21.15.114:0/1132864553 >> 172.21.15.114:0/1132864553 conn(0x7fc1640b35e0 msgr2=0x7fc164139c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:13.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.872+0000 7fc168be8640 1 -- 172.21.15.114:0/1132864553 shutdown_connections 2024-03-29T10:42:13.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.872+0000 7fc168be8640 1 -- 172.21.15.114:0/1132864553 wait complete. 2024-03-29T10:42:13.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:13.874+0000 7fc168be8640 1 librados: shutdown 2024-03-29T10:42:13.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:20.683 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:20.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.851+0000 7fe031715640 10 monclient: get_monmap_and_config 2024-03-29T10:42:20.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.851+0000 7fe031715640 10 monclient: build_initial_monmap 2024-03-29T10:42:20.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.851+0000 7fe031715640 10 monclient: monmap: 2024-03-29T10:42:20.835 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:20.835 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:20.835 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:42:20.852514+0000 2024-03-29T10:42:20.836 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:42:20.852514+0000 2024-03-29T10:42:20.836 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:20.836 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:20.836 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:20.836 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:42:20.836 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:42:20.836 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:20.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.852+0000 7fe031715640 1 Processor -- start 2024-03-29T10:42:20.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.852+0000 7fe031715640 1 -- start start 2024-03-29T10:42:20.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.852+0000 7fe031715640 10 monclient: init 2024-03-29T10:42:20.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.853+0000 7fe031715640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:20.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.853+0000 7fe031715640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:20.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.853+0000 7fe031715640 10 monclient(hunting): picked mon.noname-b con 0x7fe02c0a9750 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:20.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.853+0000 7fe031715640 10 monclient(hunting): picked mon.noname-c con 0x7fe02c0adca0 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:20.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.853+0000 7fe031715640 10 monclient(hunting): picked mon.noname-a con 0x7fe02c146a80 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:20.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.853+0000 7fe031715640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c0abdc0 con 0x7fe02c146a80 2024-03-29T10:42:20.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.853+0000 7fe031715640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c0b0070 con 0x7fe02c0adca0 2024-03-29T10:42:20.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.853+0000 7fe031715640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c0af880 con 0x7fe02c0a9750 2024-03-29T10:42:20.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.853+0000 7fe031715640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:20.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.853+0000 7fe031715640 10 monclient(hunting): authenticate will time out at 1006.094441s 2024-03-29T10:42:20.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.857+0000 7fe030f14640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe02c146a80 0x7fe02c151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44511/0 (socket says 172.21.15.114:44511) 2024-03-29T10:42:20.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.857+0000 7fe030f14640 1 -- 172.21.15.114:0/369873281 learned_addr learned my addr 172.21.15.114:0/369873281 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:20.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.857+0000 7fe02affd640 1 -- 172.21.15.114:0/369873281 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230310485 0 0) 0x7fe02c0abdc0 con 0x7fe02c146a80 2024-03-29T10:42:20.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:20.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:20.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:20.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 10 monclient(hunting): my global_id is 4350 2024-03-29T10:42:20.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 1 -- 172.21.15.114:0/369873281 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe014003520 con 0x7fe02c146a80 2024-03-29T10:42:20.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 1 -- 172.21.15.114:0/369873281 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2089698069 0 0) 0x7fe014003520 con 0x7fe02c146a80 2024-03-29T10:42:20.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 1 -- 172.21.15.114:0/369873281 >> v1:172.21.15.114:6790/0 conn(0x7fe02c0a9750 legacy=0x7fe02c143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:20.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 1 -- 172.21.15.114:0/369873281 >> v1:172.21.15.176:6789/0 conn(0x7fe02c0adca0 legacy=0x7fe02c146370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 1 monclient: found mon.noname-a 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 1 -- 172.21.15.114:0/369873281 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c0ac030 con 0x7fe02c146a80 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 10 monclient: _finish_auth 0 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 1 -- 172.21.15.114:0/369873281 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe020001690 con 0x7fe02c146a80 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe02affd640 10 monclient: dump: 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:20.845 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe031715640 5 monclient: authenticate success, global_id 4350 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.858+0000 7fe031715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.860+0000 7fe02affd640 1 -- 172.21.15.114:0/369873281 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe020001860 con 0x7fe02c146a80 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.860+0000 7fe02affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.860+0000 7fe02affd640 1 -- 172.21.15.114:0/369873281 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe020001cb0 con 0x7fe02c146a80 2024-03-29T10:42:20.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.860+0000 7fe02affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.860+0000 7fe02affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.860+0000 7fe02affd640 10 monclient: dump: 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.860+0000 7fe031715640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:20.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.860+0000 7fe031715640 4 set_mon_vals no callback set 2024-03-29T10:42:20.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.861+0000 7fe031715640 10 monclient: shutdown 2024-03-29T10:42:20.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.861+0000 7fe031715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:20.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.861+0000 7fe031715640 1 -- 172.21.15.114:0/369873281 >> v1:172.21.15.114:6789/0 conn(0x7fe02c146a80 legacy=0x7fe02c151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:20.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.861+0000 7fe031715640 1 -- 172.21.15.114:0/369873281 shutdown_connections 2024-03-29T10:42:20.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.861+0000 7fe031715640 1 -- 172.21.15.114:0/369873281 >> 172.21.15.114:0/369873281 conn(0x7fe02c0673e0 msgr2=0x7fe02c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:20.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.864+0000 7fe031715640 1 -- 172.21.15.114:0/369873281 shutdown_connections 2024-03-29T10:42:20.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.864+0000 7fe031715640 1 -- 172.21.15.114:0/369873281 wait complete. 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.865+0000 7fe031715640 10 monclient: build_initial_monmap 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.865+0000 7fe031715640 10 monclient: monmap: 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:42:20.850 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:20.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.866+0000 7fe031715640 1 librados: starting msgr at 2024-03-29T10:42:20.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.866+0000 7fe031715640 1 librados: starting objecter 2024-03-29T10:42:20.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.866+0000 7fe031715640 1 Processor -- start 2024-03-29T10:42:20.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.866+0000 7fe031715640 1 -- start start 2024-03-29T10:42:20.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.866+0000 7fe031715640 1 librados: setting wanted keys 2024-03-29T10:42:20.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.866+0000 7fe031715640 1 librados: calling monclient init 2024-03-29T10:42:20.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.866+0000 7fe031715640 10 monclient: init 2024-03-29T10:42:20.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.867+0000 7fe031715640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:20.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.867+0000 7fe031715640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:20.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.867+0000 7fe031715640 10 monclient(hunting): picked mon.noname-c con 0x7fe02c0a9750 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:20.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.867+0000 7fe031715640 10 monclient(hunting): picked mon.noname-b con 0x7fe02c0adca0 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:20.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.867+0000 7fe031715640 10 monclient(hunting): picked mon.noname-a con 0x7fe02c146a80 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:20.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.867+0000 7fe031715640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c1debc0 con 0x7fe02c146a80 2024-03-29T10:42:20.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.867+0000 7fe031715640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c1d15c0 con 0x7fe02c0adca0 2024-03-29T10:42:20.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.867+0000 7fe031715640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c1450b0 con 0x7fe02c0a9750 2024-03-29T10:42:20.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.867+0000 7fe031715640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:20.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.867+0000 7fe031715640 10 monclient(hunting): authenticate will time out at 1006.108371s 2024-03-29T10:42:20.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.874+0000 7fe030f14640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe02c146a80 0x7fe02c1d08d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44749/0 (socket says 172.21.15.114:44749) 2024-03-29T10:42:20.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.874+0000 7fe030f14640 1 -- 172.21.15.114:0/2325433930 learned_addr learned my addr 172.21.15.114:0/2325433930 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:20.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.874+0000 7fe028ff9640 10 client.?.objecter ms_handle_connect 0x7fe02c146a80 2024-03-29T10:42:20.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.874+0000 7fe028ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:20.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.874+0000 7fe028ff9640 10 client.?.objecter ms_handle_connect 0x7fe02c0a9750 2024-03-29T10:42:20.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.874+0000 7fe028ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:20.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375976860 0 0) 0x7fe02c1450b0 con 0x7fe02c0a9750 2024-03-29T10:42:20.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:20.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:20.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:20.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe018004eb0 con 0x7fe02c0a9750 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2299742243 0 0) 0x7fe018004eb0 con 0x7fe02c0a9750 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe018005300 con 0x7fe02c0a9750 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 10 client.?.objecter ms_handle_connect 0x7fe02c0adca0 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe024001bb0 con 0x7fe02c0a9750 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.875+0000 7fe028ff9640 10 monclient(hunting): dump: 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:20.865 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35665987 0 0) 0x7fe02c1debc0 con 0x7fe02c146a80 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe018005850 con 0x7fe02c146a80 2024-03-29T10:42:20.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287571101 0 0) 0x7fe02c1d15c0 con 0x7fe02c0adca0 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe02c1debc0 con 0x7fe02c0adca0 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4226736779 0 0) 0x7fe018005850 con 0x7fe02c146a80 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe018005e50 con 0x7fe02c146a80 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe020001b70 con 0x7fe02c146a80 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 10 monclient(hunting): dump: 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:20.867 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3624408866 0 0) 0x7fe018005e50 con 0x7fe02c146a80 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.876+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 >> v1:172.21.15.114:6790/0 conn(0x7fe02c0a9750 legacy=0x7fe02c1df0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 >> v1:172.21.15.176:6789/0 conn(0x7fe02c0adca0 legacy=0x7fe02c1d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe028ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe028ff9640 1 monclient: found mon.a 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe028ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe028ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:20.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c145240 con 0x7fe02c146a80 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe028ff9640 10 monclient: _finish_auth 0 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe028ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe028ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe031715640 5 monclient: authenticate success, global_id 4353 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe031715640 10 monclient: _renew_subs 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe031715640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe031715640 1 -- 172.21.15.114:0/2325433930 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c1df7f0 con 0x7fe02c146a80 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe031715640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe031715640 10 monclient: _renew_subs 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe031715640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe031715640 1 -- 172.21.15.114:0/2325433930 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c1e7300 con 0x7fe02c146a80 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.877+0000 7fe031715640 1 librados: init done 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe020001d70 con 0x7fe02c146a80 2024-03-29T10:42:20.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe028ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe020004c30 con 0x7fe02c146a80 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe028ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe028ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe028ff9640 10 monclient: dump: 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:20.870 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:20.871 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:20.871 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:20.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe031715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:20.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe031715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:20.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe031715640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:20.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe031715640 1 -- 172.21.15.114:0/2325433930 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02c04f730 con 0x7fe02c146a80 2024-03-29T10:42:20.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fe02001ac00 con 0x7fe02c146a80 2024-03-29T10:42:20.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.879+0000 7fe029ffb640 4 set_mon_vals no callback set 2024-03-29T10:42:20.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.881+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fe020084a10 con 0x7fe02c146a80 2024-03-29T10:42:20.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.881+0000 7fe028ff9640 10 client.4353.objecter ms_dispatch 0x7fe02c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:20.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.881+0000 7fe028ff9640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:20.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.881+0000 7fe028ff9640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:20.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.881+0000 7fe028ff9640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T10:42:20.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.881+0000 7fe02bfff640 10 monclient: get_auth_request con 0x7fe01806e5b0 auth_method 0 2024-03-29T10:42:20.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.882+0000 7fe028ff9640 10 client.4353.objecter ms_handle_connect 0x7fe01806e5b0 2024-03-29T10:42:20.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.889+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe020050470 con 0x7fe02c146a80 2024-03-29T10:42:20.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.889+0000 7fe028ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:20.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:20.889+0000 7fe028ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:21.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.226+0000 7fe031715640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:21.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.226+0000 7fe031715640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:21.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.226+0000 7fe031715640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:21.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.226+0000 7fe031715640 1 -- 172.21.15.114:0/2325433930 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe02c0ae8e0 con 0x7fe02c146a80 2024-03-29T10:42:21.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.228+0000 7fe028ff9640 1 -- 172.21.15.114:0/2325433930 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fe0200548e0 con 0x7fe02c146a80 2024-03-29T10:42:21.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.228+0000 7fe028ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:21.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.228+0000 7fe028ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:21.212 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:42:21.213 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.231+0000 7fe031715640 10 librados: watch_flush enter 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 10 librados: watch_flush exit 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 10 client.4353.objecter successfully canceled tick 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 1 -- 172.21.15.114:0/2325433930 >> v1:172.21.15.114:6816/775044307 conn(0x7fe01806e5b0 legacy=0x7fe018070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 10 monclient: shutdown 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 1 -- 172.21.15.114:0/2325433930 >> v1:172.21.15.114:6789/0 conn(0x7fe02c146a80 legacy=0x7fe02c1d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 1 -- 172.21.15.114:0/2325433930 shutdown_connections 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 1 -- 172.21.15.114:0/2325433930 >> 172.21.15.114:0/2325433930 conn(0x7fe02c0673e0 msgr2=0x7fe02c0aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 1 -- 172.21.15.114:0/2325433930 shutdown_connections 2024-03-29T10:42:21.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 1 -- 172.21.15.114:0/2325433930 wait complete. 2024-03-29T10:42:21.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:21.232+0000 7fe031715640 1 librados: shutdown 2024-03-29T10:42:21.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:28.042 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:28.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.204+0000 7fa512a2a640 10 monclient: get_monmap_and_config 2024-03-29T10:42:28.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.204+0000 7fa512a2a640 10 monclient: build_initial_monmap 2024-03-29T10:42:28.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.204+0000 7fa512a2a640 10 monclient: monmap: 2024-03-29T10:42:28.188 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:28.188 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:28.188 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:42:28.206112+0000 2024-03-29T10:42:28.189 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:42:28.206112+0000 2024-03-29T10:42:28.189 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:28.189 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:28.189 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:28.189 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:42:28.189 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:42:28.189 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:28.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.208+0000 7fa512a2a640 1 Processor -- start 2024-03-29T10:42:28.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.209+0000 7fa512a2a640 1 -- start start 2024-03-29T10:42:28.198 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.209+0000 7fa512a2a640 10 monclient: init 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.210+0000 7fa512a2a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.210+0000 7fa512a2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.210+0000 7fa512a2a640 10 monclient(hunting): picked mon.noname-c con 0x7fa50c1422b0 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.210+0000 7fa512a2a640 10 monclient(hunting): picked mon.noname-a con 0x7fa50c142690 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.210+0000 7fa512a2a640 10 monclient(hunting): picked mon.noname-b con 0x7fa50c0b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.210+0000 7fa512a2a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50c142c80 con 0x7fa50c142690 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.210+0000 7fa512a2a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50c0aae40 con 0x7fa50c1422b0 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.210+0000 7fa512a2a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50c13fbd0 con 0x7fa50c0b17d0 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.210+0000 7fa512a2a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.210+0000 7fa512a2a640 10 monclient(hunting): authenticate will time out at 1013.451825s 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.214+0000 7fa510fc7640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fa50c0b17d0 0x7fa50c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:58193/0 (socket says 172.21.15.114:58193) 2024-03-29T10:42:28.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.214+0000 7fa510fc7640 1 -- 172.21.15.114:0/1845998039 learned_addr learned my addr 172.21.15.114:0/1845998039 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:28.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 1 -- 172.21.15.114:0/1845998039 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554074915 0 0) 0x7fa50c0aae40 con 0x7fa50c1422b0 2024-03-29T10:42:28.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:28.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:28.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:28.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:42:28.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 1 -- 172.21.15.114:0/1845998039 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f4003520 con 0x7fa50c1422b0 2024-03-29T10:42:28.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 1 -- 172.21.15.114:0/1845998039 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219241144 0 0) 0x7fa50c13fbd0 con 0x7fa50c0b17d0 2024-03-29T10:42:28.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:28.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:28.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:28.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:42:28.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.216+0000 7fa50affd640 1 -- 172.21.15.114:0/1845998039 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa50c0aae40 con 0x7fa50c0b17d0 2024-03-29T10:42:28.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.217+0000 7fa50affd640 1 -- 172.21.15.114:0/1845998039 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2905551867 0 0) 0x7fa50c0aae40 con 0x7fa50c0b17d0 2024-03-29T10:42:28.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.217+0000 7fa50affd640 1 -- 172.21.15.114:0/1845998039 >> v1:172.21.15.176:6789/0 conn(0x7fa50c1422b0 legacy=0x7fa50c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:28.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.217+0000 7fa50affd640 1 -- 172.21.15.114:0/1845998039 >> v1:172.21.15.114:6789/0 conn(0x7fa50c142690 legacy=0x7fa50c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:28.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:28.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 1 monclient: found mon.noname-b 2024-03-29T10:42:28.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:28.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:42:28.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 1 -- 172.21.15.114:0/1845998039 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c151940 con 0x7fa50c0b17d0 2024-03-29T10:42:28.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 10 monclient: _finish_auth 0 2024-03-29T10:42:28.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:28.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 1 -- 172.21.15.114:0/1845998039 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa504001970 con 0x7fa50c0b17d0 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 10 monclient: dump: 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:28.205 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:28.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa512a2a640 5 monclient: authenticate success, global_id 4358 2024-03-29T10:42:28.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa512a2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:28.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 1 -- 172.21.15.114:0/1845998039 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa504001ad0 con 0x7fa50c0b17d0 2024-03-29T10:42:28.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:28.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 1 -- 172.21.15.114:0/1845998039 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa504001e60 con 0x7fa50c0b17d0 2024-03-29T10:42:28.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:28.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:28.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa50affd640 10 monclient: dump: 2024-03-29T10:42:28.206 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:28.206 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:28.206 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:28.207 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:28.207 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:28.207 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:28.207 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:28.207 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:28.207 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:28.207 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:28.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa512a2a640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:28.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.218+0000 7fa512a2a640 4 set_mon_vals no callback set 2024-03-29T10:42:28.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.219+0000 7fa512a2a640 10 monclient: shutdown 2024-03-29T10:42:28.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.219+0000 7fa512a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:28.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.219+0000 7fa512a2a640 1 -- 172.21.15.114:0/1845998039 >> v1:172.21.15.114:6790/0 conn(0x7fa50c0b17d0 legacy=0x7fa50c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:28.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.220+0000 7fa512a2a640 1 -- 172.21.15.114:0/1845998039 shutdown_connections 2024-03-29T10:42:28.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.220+0000 7fa512a2a640 1 -- 172.21.15.114:0/1845998039 >> 172.21.15.114:0/1845998039 conn(0x7fa50c066a00 msgr2=0x7fa50c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:28.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.221+0000 7fa512a2a640 1 -- 172.21.15.114:0/1845998039 shutdown_connections 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.221+0000 7fa512a2a640 1 -- 172.21.15.114:0/1845998039 wait complete. 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.222+0000 7fa512a2a640 10 monclient: build_initial_monmap 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.222+0000 7fa512a2a640 10 monclient: monmap: 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:42:28.210 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.222+0000 7fa512a2a640 1 librados: starting msgr at 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.223+0000 7fa512a2a640 1 librados: starting objecter 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.223+0000 7fa512a2a640 1 Processor -- start 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.223+0000 7fa512a2a640 1 -- start start 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.223+0000 7fa512a2a640 1 librados: setting wanted keys 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.223+0000 7fa512a2a640 1 librados: calling monclient init 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.223+0000 7fa512a2a640 10 monclient: init 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.224+0000 7fa512a2a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.224+0000 7fa512a2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.224+0000 7fa512a2a640 10 monclient(hunting): picked mon.noname-c con 0x7fa50c0b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.224+0000 7fa512a2a640 10 monclient(hunting): picked mon.noname-a con 0x7fa50c1422b0 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.224+0000 7fa512a2a640 10 monclient(hunting): picked mon.noname-b con 0x7fa50c142690 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.224+0000 7fa512a2a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50c0ac8c0 con 0x7fa50c1422b0 2024-03-29T10:42:28.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.224+0000 7fa512a2a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50c0acb30 con 0x7fa50c142690 2024-03-29T10:42:28.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.224+0000 7fa512a2a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50c0ace10 con 0x7fa50c0b17d0 2024-03-29T10:42:28.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.224+0000 7fa512a2a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:28.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.224+0000 7fa512a2a640 10 monclient(hunting): authenticate will time out at 1013.465505s 2024-03-29T10:42:28.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.228+0000 7fa50bfff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fa50c0b17d0 0x7fa50c150d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:58277/0 (socket says 172.21.15.114:58277) 2024-03-29T10:42:28.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.228+0000 7fa50bfff640 1 -- 172.21.15.114:0/1542614756 learned_addr learned my addr 172.21.15.114:0/1542614756 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:28.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.231+0000 7fa508ff9640 10 client.?.objecter ms_handle_connect 0x7fa50c0b17d0 2024-03-29T10:42:28.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.231+0000 7fa508ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:28.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.231+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654094478 0 0) 0x7fa50c0ace10 con 0x7fa50c0b17d0 2024-03-29T10:42:28.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.231+0000 7fa508ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:28.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.231+0000 7fa508ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:28.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.231+0000 7fa508ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:28.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.231+0000 7fa508ff9640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:42:28.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.231+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f8004eb0 con 0x7fa50c0b17d0 2024-03-29T10:42:28.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.231+0000 7fa508ff9640 10 client.?.objecter ms_handle_connect 0x7fa50c1422b0 2024-03-29T10:42:28.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.231+0000 7fa508ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:28.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.233+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615834216 0 0) 0x7fa50c0ac8c0 con 0x7fa50c1422b0 2024-03-29T10:42:28.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.233+0000 7fa508ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:28.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.233+0000 7fa508ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:28.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.233+0000 7fa508ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:28.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.233+0000 7fa508ff9640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.233+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa50c0ace10 con 0x7fa50c1422b0 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 620467421 0 0) 0x7fa50c0ace10 con 0x7fa50c1422b0 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4f8005490 con 0x7fa50c1422b0 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa500001770 con 0x7fa50c1422b0 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 10 monclient(hunting): dump: 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:28.219 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2744350443 0 0) 0x7fa4f8004eb0 con 0x7fa50c0b17d0 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4f8005b60 con 0x7fa50c0b17d0 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa4fc001a90 con 0x7fa50c0b17d0 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 10 monclient(hunting): dump: 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:28.220 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1460312260 0 0) 0x7fa4f8005490 con 0x7fa50c1422b0 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 >> v1:172.21.15.114:6790/0 conn(0x7fa50c0b17d0 legacy=0x7fa50c150d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 >> v1:172.21.15.176:6789/0 conn(0x7fa50c142690 legacy=0x7fa50c1e1c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 1 monclient: found mon.a 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c131d70 con 0x7fa50c1422b0 2024-03-29T10:42:28.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 10 monclient: _finish_auth 0 2024-03-29T10:42:28.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:42:28.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa508ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:28.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa512a2a640 5 monclient: authenticate success, global_id 4356 2024-03-29T10:42:28.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa512a2a640 10 monclient: _renew_subs 2024-03-29T10:42:28.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa512a2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:28.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa512a2a640 1 -- 172.21.15.114:0/1542614756 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50c1327a0 con 0x7fa50c1422b0 2024-03-29T10:42:28.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa512a2a640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:28.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa512a2a640 10 monclient: _renew_subs 2024-03-29T10:42:28.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa512a2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:28.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa512a2a640 1 -- 172.21.15.114:0/1542614756 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50c0aba90 con 0x7fa50c1422b0 2024-03-29T10:42:28.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.234+0000 7fa512a2a640 1 librados: init done 2024-03-29T10:42:28.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.235+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa500001c60 con 0x7fa50c1422b0 2024-03-29T10:42:28.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.235+0000 7fa508ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:28.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.235+0000 7fa509ffb640 4 set_mon_vals no callback set 2024-03-29T10:42:28.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.235+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa500001770 con 0x7fa50c1422b0 2024-03-29T10:42:28.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.235+0000 7fa508ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:28.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.235+0000 7fa508ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:28.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.235+0000 7fa508ff9640 10 monclient: dump: 2024-03-29T10:42:28.223 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:28.223 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:28.223 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:28.223 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:28.224 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:28.224 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:28.224 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:28.224 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:28.224 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:28.224 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:28.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.240+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fa500019460 con 0x7fa50c1422b0 2024-03-29T10:42:28.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.241+0000 7fa512a2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:28.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.241+0000 7fa512a2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:28.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.241+0000 7fa512a2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:28.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.241+0000 7fa512a2a640 1 -- 172.21.15.114:0/1542614756 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ec005080 con 0x7fa50c1422b0 2024-03-29T10:42:28.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.242+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fa500083240 con 0x7fa50c1422b0 2024-03-29T10:42:28.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.242+0000 7fa508ff9640 10 client.4356.objecter ms_dispatch 0x7fa50c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:28.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.242+0000 7fa508ff9640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:28.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.242+0000 7fa508ff9640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:28.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.242+0000 7fa508ff9640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-29T10:42:28.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.243+0000 7fa50bfff640 10 monclient: get_auth_request con 0x7fa4f806e720 auth_method 0 2024-03-29T10:42:28.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.243+0000 7fa508ff9640 10 client.4356.objecter ms_handle_connect 0x7fa4f806e720 2024-03-29T10:42:28.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.250+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa50004eb80 con 0x7fa50c1422b0 2024-03-29T10:42:28.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.250+0000 7fa508ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:28.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.250+0000 7fa508ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:28.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.575+0000 7fa512a2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:28.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.575+0000 7fa512a2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:28.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.575+0000 7fa512a2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:28.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.575+0000 7fa512a2a640 1 -- 172.21.15.114:0/1542614756 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4ec005330 con 0x7fa50c1422b0 2024-03-29T10:42:28.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.576+0000 7fa508ff9640 1 -- 172.21.15.114:0/1542614756 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fa50004eb80 con 0x7fa50c1422b0 2024-03-29T10:42:28.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.576+0000 7fa508ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:28.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.576+0000 7fa508ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:28.560 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:42:28.561 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:28.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.580+0000 7fa512a2a640 10 librados: watch_flush enter 2024-03-29T10:42:28.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.580+0000 7fa512a2a640 10 librados: watch_flush exit 2024-03-29T10:42:28.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.580+0000 7fa512a2a640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-29T10:42:28.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.580+0000 7fa512a2a640 10 client.4356.objecter successfully canceled tick 2024-03-29T10:42:28.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.580+0000 7fa512a2a640 1 -- 172.21.15.114:0/1542614756 >> v1:172.21.15.114:6816/775044307 conn(0x7fa4f806e720 legacy=0x7fa4f8070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:28.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.580+0000 7fa512a2a640 10 monclient: shutdown 2024-03-29T10:42:28.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.580+0000 7fa512a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:28.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.580+0000 7fa512a2a640 1 -- 172.21.15.114:0/1542614756 >> v1:172.21.15.114:6789/0 conn(0x7fa50c1422b0 legacy=0x7fa50c12e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:28.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.580+0000 7fa512a2a640 1 -- 172.21.15.114:0/1542614756 shutdown_connections 2024-03-29T10:42:28.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.580+0000 7fa512a2a640 1 -- 172.21.15.114:0/1542614756 >> 172.21.15.114:0/1542614756 conn(0x7fa50c066a00 msgr2=0x7fa50c0b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:28.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.581+0000 7fa512a2a640 1 -- 172.21.15.114:0/1542614756 shutdown_connections 2024-03-29T10:42:28.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.581+0000 7fa512a2a640 1 -- 172.21.15.114:0/1542614756 wait complete. 2024-03-29T10:42:28.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:28.581+0000 7fa512a2a640 1 librados: shutdown 2024-03-29T10:42:28.580 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:35.387 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:35.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.564+0000 7faf6da0f640 10 monclient: get_monmap_and_config 2024-03-29T10:42:35.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.564+0000 7faf6da0f640 10 monclient: build_initial_monmap 2024-03-29T10:42:35.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.564+0000 7faf6da0f640 10 monclient: monmap: 2024-03-29T10:42:35.548 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:35.548 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:35.548 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:42:35.565773+0000 2024-03-29T10:42:35.548 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:42:35.565773+0000 2024-03-29T10:42:35.548 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:35.549 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:35.549 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:35.549 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:42:35.549 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:42:35.549 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:35.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.566+0000 7faf6da0f640 1 Processor -- start 2024-03-29T10:42:35.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.566+0000 7faf6da0f640 1 -- start start 2024-03-29T10:42:35.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.566+0000 7faf6da0f640 10 monclient: init 2024-03-29T10:42:35.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.567+0000 7faf6da0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:35.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.567+0000 7faf6da0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:35.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.567+0000 7faf6da0f640 10 monclient(hunting): picked mon.noname-b con 0x7faf680b4330 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:35.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.567+0000 7faf6da0f640 10 monclient(hunting): picked mon.noname-c con 0x7faf681470b0 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:35.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.567+0000 7faf6da0f640 10 monclient(hunting): picked mon.noname-a con 0x7faf681439b0 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:35.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.567+0000 7faf6da0f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6806f2a0 con 0x7faf681439b0 2024-03-29T10:42:35.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.567+0000 7faf6da0f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf68146b90 con 0x7faf681470b0 2024-03-29T10:42:35.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.567+0000 7faf6da0f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf680b2ca0 con 0x7faf680b4330 2024-03-29T10:42:35.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.567+0000 7faf6da0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:35.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.567+0000 7faf6da0f640 10 monclient(hunting): authenticate will time out at 1020.808713s 2024-03-29T10:42:35.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.571+0000 7faf66ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7faf680b4330 0x7faf680b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:51072/0 (socket says 172.21.15.114:51072) 2024-03-29T10:42:35.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.571+0000 7faf66ffd640 1 -- 172.21.15.114:0/152428189 learned_addr learned my addr 172.21.15.114:0/152428189 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:35.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.572+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137472493 0 0) 0x7faf680b2ca0 con 0x7faf680b4330 2024-03-29T10:42:35.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.572+0000 7faf65ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:35.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.572+0000 7faf65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:35.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.572+0000 7faf65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:35.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.572+0000 7faf65ffb640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:42:35.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.572+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf50003520 con 0x7faf680b4330 2024-03-29T10:42:35.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.573+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201567918 0 0) 0x7faf68146b90 con 0x7faf681470b0 2024-03-29T10:42:35.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.573+0000 7faf65ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:35.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.573+0000 7faf65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:35.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.573+0000 7faf65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.573+0000 7faf65ffb640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.573+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf680b2ca0 con 0x7faf681470b0 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856116827 0 0) 0x7faf6806f2a0 con 0x7faf681439b0 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient(hunting): my global_id is 4359 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf68146b90 con 0x7faf681439b0 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566698441 0 0) 0x7faf680b2ca0 con 0x7faf681470b0 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 >> v1:172.21.15.114:6790/0 conn(0x7faf680b4330 legacy=0x7faf680b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 >> v1:172.21.15.114:6789/0 conn(0x7faf681439b0 legacy=0x7faf68151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:35.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 1 monclient: found mon.noname-c 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6806e0c0 con 0x7faf681470b0 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7faf5c001670 con 0x7faf681470b0 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient: dump: 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.560 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf6da0f640 5 monclient: authenticate success, global_id 4330 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf6da0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf5c0017d0 con 0x7faf681470b0 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 1 -- 172.21.15.114:0/152428189 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7faf5c001be0 con 0x7faf681470b0 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf65ffb640 10 monclient: dump: 2024-03-29T10:42:35.561 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:35.562 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:35.562 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.562 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.562 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:35.562 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:35.562 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:35.562 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:35.562 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:35.562 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:35.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf6da0f640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:35.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.574+0000 7faf6da0f640 4 set_mon_vals no callback set 2024-03-29T10:42:35.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.575+0000 7faf6da0f640 10 monclient: shutdown 2024-03-29T10:42:35.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.575+0000 7faf6da0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:35.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.575+0000 7faf6da0f640 1 -- 172.21.15.114:0/152428189 >> v1:172.21.15.176:6789/0 conn(0x7faf681470b0 legacy=0x7faf680b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.575+0000 7faf6da0f640 1 -- 172.21.15.114:0/152428189 shutdown_connections 2024-03-29T10:42:35.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.575+0000 7faf6da0f640 1 -- 172.21.15.114:0/152428189 >> 172.21.15.114:0/152428189 conn(0x7faf68066a00 msgr2=0x7faf68066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:35.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.580+0000 7faf6da0f640 1 -- 172.21.15.114:0/152428189 shutdown_connections 2024-03-29T10:42:35.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.580+0000 7faf6da0f640 1 -- 172.21.15.114:0/152428189 wait complete. 2024-03-29T10:42:35.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.583+0000 7faf6da0f640 10 monclient: build_initial_monmap 2024-03-29T10:42:35.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.584+0000 7faf6da0f640 10 monclient: monmap: 2024-03-29T10:42:35.567 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:35.567 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:35.567 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:42:35.567 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:42:35.567 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:35.567 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:35.567 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:35.567 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:42:35.568 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:42:35.568 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:35.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.584+0000 7faf6da0f640 1 librados: starting msgr at 2024-03-29T10:42:35.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.584+0000 7faf6da0f640 1 librados: starting objecter 2024-03-29T10:42:35.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.584+0000 7faf6da0f640 1 Processor -- start 2024-03-29T10:42:35.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.584+0000 7faf6da0f640 1 -- start start 2024-03-29T10:42:35.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.584+0000 7faf6da0f640 1 librados: setting wanted keys 2024-03-29T10:42:35.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.584+0000 7faf6da0f640 1 librados: calling monclient init 2024-03-29T10:42:35.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.584+0000 7faf6da0f640 10 monclient: init 2024-03-29T10:42:35.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.585+0000 7faf6da0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:35.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.585+0000 7faf6da0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:35.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.585+0000 7faf6da0f640 10 monclient(hunting): picked mon.noname-a con 0x7faf680b4330 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:35.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.586+0000 7faf6da0f640 10 monclient(hunting): picked mon.noname-b con 0x7faf681439b0 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:35.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.588+0000 7faf6da0f640 10 monclient(hunting): picked mon.noname-c con 0x7faf681470b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:35.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.588+0000 7faf6da0f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf68120a10 con 0x7faf680b4330 2024-03-29T10:42:35.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.588+0000 7faf6da0f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf681216d0 con 0x7faf681439b0 2024-03-29T10:42:35.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.588+0000 7faf6da0f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf681219b0 con 0x7faf681470b0 2024-03-29T10:42:35.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.588+0000 7faf6da0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:35.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.588+0000 7faf6da0f640 10 monclient(hunting): authenticate will time out at 1020.829937s 2024-03-29T10:42:35.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.593+0000 7faf677fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7faf681470b0 0x7faf681dee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:52798/0 (socket says 172.21.15.114:52798) 2024-03-29T10:42:35.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.593+0000 7faf677fe640 1 -- 172.21.15.114:0/654761751 learned_addr learned my addr 172.21.15.114:0/654761751 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:35.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.593+0000 7faf4ffff640 10 client.?.objecter ms_handle_connect 0x7faf681470b0 2024-03-29T10:42:35.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.593+0000 7faf4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:35.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.593+0000 7faf4ffff640 10 client.?.objecter ms_handle_connect 0x7faf680b4330 2024-03-29T10:42:35.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.593+0000 7faf4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:35.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.593+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463171813 0 0) 0x7faf68120a10 con 0x7faf680b4330 2024-03-29T10:42:35.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.593+0000 7faf4ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:35.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.593+0000 7faf4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:35.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.594+0000 7faf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:35.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.594+0000 7faf4ffff640 10 monclient(hunting): my global_id is 4362 2024-03-29T10:42:35.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.594+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf54004eb0 con 0x7faf680b4330 2024-03-29T10:42:35.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 10 client.?.objecter ms_handle_connect 0x7faf681439b0 2024-03-29T10:42:35.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:35.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237660746 0 0) 0x7faf681216d0 con 0x7faf681439b0 2024-03-29T10:42:35.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:35.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:35.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:35.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 10 monclient(hunting): my global_id is 4333 2024-03-29T10:42:35.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf68120a10 con 0x7faf681439b0 2024-03-29T10:42:35.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197204958 0 0) 0x7faf681219b0 con 0x7faf681470b0 2024-03-29T10:42:35.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:35.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:35.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:35.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:42:35.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf681216d0 con 0x7faf681470b0 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.595+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1259639648 0 0) 0x7faf54004eb0 con 0x7faf680b4330 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf681219b0 con 0x7faf680b4330 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7faf60001a70 con 0x7faf680b4330 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 10 monclient(hunting): dump: 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:35.589 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3247837405 0 0) 0x7faf68120a10 con 0x7faf681439b0 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf54004eb0 con 0x7faf681439b0 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7faf5c001ae0 con 0x7faf681439b0 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.597+0000 7faf4ffff640 10 monclient(hunting): dump: 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.590 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 742953342 0 0) 0x7faf681216d0 con 0x7faf681470b0 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf68120a10 con 0x7faf681470b0 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7faf58001870 con 0x7faf681470b0 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 10 monclient(hunting): dump: 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:35.591 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1235241542 0 0) 0x7faf681219b0 con 0x7faf680b4330 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 >> v1:172.21.15.114:6790/0 conn(0x7faf681470b0 legacy=0x7faf681dee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 >> v1:172.21.15.176:6789/0 conn(0x7faf681439b0 legacy=0x7faf681406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 1 monclient: found mon.a 2024-03-29T10:42:35.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6811f9e0 con 0x7faf680b4330 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 10 monclient: _finish_auth 0 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf6da0f640 5 monclient: authenticate success, global_id 4362 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf6da0f640 10 monclient: _renew_subs 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf6da0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf6da0f640 1 -- 172.21.15.114:0/654761751 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6811fc90 con 0x7faf680b4330 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf6da0f640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf6da0f640 10 monclient: _renew_subs 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf6da0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf6da0f640 1 -- 172.21.15.114:0/654761751 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf68140fd0 con 0x7faf680b4330 2024-03-29T10:42:35.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf6da0f640 1 librados: init done 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf60001fe0 con 0x7faf680b4330 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7faf60004d00 con 0x7faf680b4330 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.598+0000 7faf4ffff640 10 monclient: dump: 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:35.594 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.599+0000 7faf657fa640 4 set_mon_vals no callback set 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.599+0000 7faf6da0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.600+0000 7faf6da0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.600+0000 7faf6da0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.600+0000 7faf6da0f640 1 -- 172.21.15.114:0/654761751 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf48005080 con 0x7faf680b4330 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.602+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7faf6001acd0 con 0x7faf680b4330 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.603+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7faf60084ca0 con 0x7faf680b4330 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.603+0000 7faf667fc640 10 monclient: get_auth_request con 0x7faf5406e0c0 auth_method 0 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.603+0000 7faf4ffff640 10 client.4362.objecter ms_dispatch 0x7faf68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.603+0000 7faf4ffff640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:35.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.603+0000 7faf4ffff640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:35.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.604+0000 7faf4ffff640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T10:42:35.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.604+0000 7faf4ffff640 10 client.4362.objecter ms_handle_connect 0x7faf5406e0c0 2024-03-29T10:42:35.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.613+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf60050580 con 0x7faf680b4330 2024-03-29T10:42:35.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.614+0000 7faf4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:35.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:35.614+0000 7faf4ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:35.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.004+0000 7faf6da0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:35.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.004+0000 7faf6da0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:35.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.004+0000 7faf6da0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:42:35.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.004+0000 7faf6da0f640 1 -- 172.21.15.114:0/654761751 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf48005330 con 0x7faf680b4330 2024-03-29T10:42:35.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.005+0000 7faf4ffff640 1 -- 172.21.15.114:0/654761751 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7faf600524b0 con 0x7faf680b4330 2024-03-29T10:42:35.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.005+0000 7faf4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:35.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.005+0000 7faf4ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:35.989 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:42:35.989 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:35.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.009+0000 7faf6da0f640 10 librados: watch_flush enter 2024-03-29T10:42:35.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.009+0000 7faf6da0f640 10 librados: watch_flush exit 2024-03-29T10:42:35.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.009+0000 7faf6da0f640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T10:42:35.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.009+0000 7faf6da0f640 10 client.4362.objecter successfully canceled tick 2024-03-29T10:42:35.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.009+0000 7faf6da0f640 1 -- 172.21.15.114:0/654761751 >> v1:172.21.15.114:6816/775044307 conn(0x7faf5406e0c0 legacy=0x7faf54070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.009+0000 7faf6da0f640 10 monclient: shutdown 2024-03-29T10:42:35.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.009+0000 7faf6da0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:35.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.009+0000 7faf6da0f640 1 -- 172.21.15.114:0/654761751 >> v1:172.21.15.114:6789/0 conn(0x7faf680b4330 legacy=0x7faf68120190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.010+0000 7faf6da0f640 1 -- 172.21.15.114:0/654761751 shutdown_connections 2024-03-29T10:42:35.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.010+0000 7faf6da0f640 1 -- 172.21.15.114:0/654761751 >> 172.21.15.114:0/654761751 conn(0x7faf68066a00 msgr2=0x7faf68142f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:35.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.011+0000 7faf6da0f640 1 -- 172.21.15.114:0/654761751 shutdown_connections 2024-03-29T10:42:35.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.011+0000 7faf6da0f640 1 -- 172.21.15.114:0/654761751 wait complete. 2024-03-29T10:42:35.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:36.012+0000 7faf6da0f640 1 librados: shutdown 2024-03-29T10:42:36.015 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:42.822 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:42.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.985+0000 7fb3ddae6640 10 monclient: get_monmap_and_config 2024-03-29T10:42:42.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.985+0000 7fb3ddae6640 10 monclient: build_initial_monmap 2024-03-29T10:42:42.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.985+0000 7fb3ddae6640 10 monclient: monmap: 2024-03-29T10:42:42.969 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:42.969 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:42.970 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:42:42.987190+0000 2024-03-29T10:42:42.970 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:42:42.987190+0000 2024-03-29T10:42:42.970 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:42.970 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:42.970 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:42.970 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:42:42.970 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:42:42.970 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:42.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 1 Processor -- start 2024-03-29T10:42:42.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 1 -- start start 2024-03-29T10:42:42.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 10 monclient: init 2024-03-29T10:42:42.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:42.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:42.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 10 monclient(hunting): picked mon.noname-c con 0x7fb3d81422b0 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:42.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 10 monclient(hunting): picked mon.noname-b con 0x7fb3d8142690 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:42.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 10 monclient(hunting): picked mon.noname-a con 0x7fb3d80b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:42.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d8142c80 con 0x7fb3d80b17d0 2024-03-29T10:42:42.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d80aae40 con 0x7fb3d81422b0 2024-03-29T10:42:42.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d813fbd0 con 0x7fb3d8142690 2024-03-29T10:42:42.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:42.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.987+0000 7fb3ddae6640 10 monclient(hunting): authenticate will time out at 1028.228969s 2024-03-29T10:42:42.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.988+0000 7fb3d7fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb3d80b17d0 0x7fb3d80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39790/0 (socket says 172.21.15.114:39790) 2024-03-29T10:42:42.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.988+0000 7fb3d7fff640 1 -- 172.21.15.114:0/3215154548 learned_addr learned my addr 172.21.15.114:0/3215154548 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:42.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822057753 0 0) 0x7fb3d80aae40 con 0x7fb3d81422b0 2024-03-29T10:42:42.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:42.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:42.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:42.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:42:42.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c0003520 con 0x7fb3d81422b0 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246518851 0 0) 0x7fb3d813fbd0 con 0x7fb3d8142690 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d80aae40 con 0x7fb3d8142690 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241933674 0 0) 0x7fb3d8142c80 con 0x7fb3d80b17d0 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient(hunting): my global_id is 4365 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d813fbd0 con 0x7fb3d80b17d0 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 843552771 0 0) 0x7fb3c0003520 con 0x7fb3d81422b0 2024-03-29T10:42:42.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 >> v1:172.21.15.114:6790/0 conn(0x7fb3d8142690 legacy=0x7fb3d80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 >> v1:172.21.15.114:6789/0 conn(0x7fb3d80b17d0 legacy=0x7fb3d80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 monclient: found mon.noname-c 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d8151940 con 0x7fb3d81422b0 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3ddae6640 5 monclient: authenticate success, global_id 4336 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3ddae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb3c8001690 con 0x7fb3d81422b0 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:42.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3d67fc640 10 monclient: dump: 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.989+0000 7fb3ddae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3c80018b0 con 0x7fb3d81422b0 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:42.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3ddae6640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3ddae6640 4 set_mon_vals no callback set 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3d67fc640 1 -- 172.21.15.114:0/3215154548 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb3c8001bc0 con 0x7fb3d81422b0 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3d67fc640 10 monclient: dump: 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:42.978 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:42.979 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:42.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3ddae6640 10 monclient: shutdown 2024-03-29T10:42:42.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3ddae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:42.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3215154548 >> v1:172.21.15.176:6789/0 conn(0x7fb3d81422b0 legacy=0x7fb3d813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:42.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3215154548 shutdown_connections 2024-03-29T10:42:42.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.990+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3215154548 >> 172.21.15.114:0/3215154548 conn(0x7fb3d8066a00 msgr2=0x7fb3d8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:42.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.991+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3215154548 shutdown_connections 2024-03-29T10:42:42.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.991+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3215154548 wait complete. 2024-03-29T10:42:42.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.992+0000 7fb3ddae6640 10 monclient: build_initial_monmap 2024-03-29T10:42:42.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.992+0000 7fb3ddae6640 10 monclient: monmap: 2024-03-29T10:42:42.982 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:42.982 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:42.982 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:42:42.982 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:42:42.982 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:42.982 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:42.982 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.992+0000 7fb3ddae6640 1 librados: starting msgr at 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.992+0000 7fb3ddae6640 1 librados: starting objecter 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.992+0000 7fb3ddae6640 1 Processor -- start 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.993+0000 7fb3ddae6640 1 -- start start 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.993+0000 7fb3ddae6640 1 librados: setting wanted keys 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.993+0000 7fb3ddae6640 1 librados: calling monclient init 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.993+0000 7fb3ddae6640 10 monclient: init 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.993+0000 7fb3ddae6640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3ddae6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3ddae6640 10 monclient(hunting): picked mon.noname-b con 0x7fb3d80b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:42.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3ddae6640 10 monclient(hunting): picked mon.noname-c con 0x7fb3d81422b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3ddae6640 10 monclient(hunting): picked mon.noname-a con 0x7fb3d8142690 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3ddae6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d813b740 con 0x7fb3d8142690 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3ddae6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d8070670 con 0x7fb3d80b17d0 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3ddae6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d806fc90 con 0x7fb3d81422b0 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3ddae6640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3ddae6640 10 monclient(hunting): authenticate will time out at 1028.235107s 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3d7fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb3d8142690 0x7fb3d813b030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39970/0 (socket says 172.21.15.114:39970) 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3d7fff640 1 -- 172.21.15.114:0/3371317225 learned_addr learned my addr 172.21.15.114:0/3371317225 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3bffff640 10 client.?.objecter ms_handle_connect 0x7fb3d8142690 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3bffff640 10 client.?.objecter ms_handle_connect 0x7fb3d80b17d0 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.994+0000 7fb3bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483834283 0 0) 0x7fb3d8070670 con 0x7fb3d80b17d0 2024-03-29T10:42:42.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c4004eb0 con 0x7fb3d80b17d0 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 client.?.objecter ms_handle_connect 0x7fb3d81422b0 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794666990 0 0) 0x7fb3d813b740 con 0x7fb3d8142690 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): my global_id is 4368 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d8070670 con 0x7fb3d8142690 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339848946 0 0) 0x7fb3d806fc90 con 0x7fb3d81422b0 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:42.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d813b740 con 0x7fb3d81422b0 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 354261265 0 0) 0x7fb3d8070670 con 0x7fb3d8142690 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.995+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3d806fc90 con 0x7fb3d8142690 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb3cc0018b0 con 0x7fb3d8142690 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient(hunting): dump: 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.986 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1816394937 0 0) 0x7fb3c4004eb0 con 0x7fb3d80b17d0 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3d8070670 con 0x7fb3d80b17d0 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb3c8001550 con 0x7fb3d80b17d0 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient(hunting): dump: 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:42.987 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 733881231 0 0) 0x7fb3d813b740 con 0x7fb3d81422b0 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3c4004eb0 con 0x7fb3d81422b0 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb3d0001ab0 con 0x7fb3d81422b0 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient(hunting): dump: 2024-03-29T10:42:42.988 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 103711964 0 0) 0x7fb3d8070670 con 0x7fb3d80b17d0 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 >> v1:172.21.15.114:6790/0 conn(0x7fb3d81422b0 legacy=0x7fb3d813a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 >> v1:172.21.15.114:6789/0 conn(0x7fb3d8142690 legacy=0x7fb3d813b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 monclient: found mon.b 2024-03-29T10:42:42.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d806ff70 con 0x7fb3d80b17d0 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient: _finish_auth 0 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3ddae6640 5 monclient: authenticate success, global_id 4339 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3ddae6640 10 monclient: _renew_subs 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3ddae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3371317225 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d8070250 con 0x7fb3d80b17d0 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3ddae6640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3ddae6640 10 monclient: _renew_subs 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3ddae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3371317225 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d813b9a0 con 0x7fb3d80b17d0 2024-03-29T10:42:42.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.996+0000 7fb3ddae6640 1 librados: init done 2024-03-29T10:42:42.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.998+0000 7fb3ddae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:42.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.998+0000 7fb3ddae6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:42.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.998+0000 7fb3ddae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:42.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.998+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3371317225 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d8075f10 con 0x7fb3d80b17d0 2024-03-29T10:42:42.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.999+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3c80038a0 con 0x7fb3d80b17d0 2024-03-29T10:42:42.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.999+0000 7fb3bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:42.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.999+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb3c8003bb0 con 0x7fb3d80b17d0 2024-03-29T10:42:42.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.999+0000 7fb3bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:42.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.999+0000 7fb3bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:42.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.999+0000 7fb3bffff640 10 monclient: dump: 2024-03-29T10:42:42.992 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:42.992 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:42.999+0000 7fb3d5ffb640 4 set_mon_vals no callback set 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.001+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fb3c8019b80 con 0x7fb3d80b17d0 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.001+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fb3c8083800 con 0x7fb3d80b17d0 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.001+0000 7fb3bffff640 10 client.4339.objecter ms_dispatch 0x7fb3d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.001+0000 7fb3bffff640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:42.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.001+0000 7fb3bffff640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:42.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.001+0000 7fb3bffff640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T10:42:42.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.002+0000 7fb3d6ffd640 10 monclient: get_auth_request con 0x7fb3c406e230 auth_method 0 2024-03-29T10:42:42.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.002+0000 7fb3bffff640 10 client.4339.objecter ms_handle_connect 0x7fb3c406e230 2024-03-29T10:42:42.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.008+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3c804f0e0 con 0x7fb3d80b17d0 2024-03-29T10:42:42.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.008+0000 7fb3bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:42.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.008+0000 7fb3bffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:43.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.376+0000 7fb3ddae6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:43.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.376+0000 7fb3ddae6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:43.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.376+0000 7fb3ddae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:43.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.376+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3371317225 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3d80b2270 con 0x7fb3d80b17d0 2024-03-29T10:42:43.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.378+0000 7fb3bffff640 1 -- 172.21.15.114:0/3371317225 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fb3c8051010 con 0x7fb3d80b17d0 2024-03-29T10:42:43.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.378+0000 7fb3bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:43.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.378+0000 7fb3bffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:43.362 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:42:43.362 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:43.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.381+0000 7fb3ddae6640 10 librados: watch_flush enter 2024-03-29T10:42:43.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.381+0000 7fb3ddae6640 10 librados: watch_flush exit 2024-03-29T10:42:43.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.381+0000 7fb3ddae6640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T10:42:43.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.381+0000 7fb3ddae6640 10 client.4339.objecter successfully canceled tick 2024-03-29T10:42:43.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.382+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3371317225 >> v1:172.21.15.114:6816/775044307 conn(0x7fb3c406e230 legacy=0x7fb3c40706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:43.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.382+0000 7fb3ddae6640 10 monclient: shutdown 2024-03-29T10:42:43.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.382+0000 7fb3ddae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:43.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.382+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3371317225 >> v1:172.21.15.176:6789/0 conn(0x7fb3d80b17d0 legacy=0x7fb3d8070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:43.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.382+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3371317225 shutdown_connections 2024-03-29T10:42:43.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.382+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3371317225 >> 172.21.15.114:0/3371317225 conn(0x7fb3d8066a00 msgr2=0x7fb3d813f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:43.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.382+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3371317225 shutdown_connections 2024-03-29T10:42:43.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.382+0000 7fb3ddae6640 1 -- 172.21.15.114:0/3371317225 wait complete. 2024-03-29T10:42:43.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:43.382+0000 7fb3ddae6640 1 librados: shutdown 2024-03-29T10:42:43.380 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:50.188 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:50.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.351+0000 7fa9ac28d640 10 monclient: get_monmap_and_config 2024-03-29T10:42:50.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.351+0000 7fa9ac28d640 10 monclient: build_initial_monmap 2024-03-29T10:42:50.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.351+0000 7fa9ac28d640 10 monclient: monmap: 2024-03-29T10:42:50.337 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:50.337 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:50.337 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:42:50.352207+0000 2024-03-29T10:42:50.338 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:42:50.352207+0000 2024-03-29T10:42:50.338 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:50.338 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:50.338 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:50.338 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:42:50.338 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:42:50.338 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:50.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.355+0000 7fa9ac28d640 1 Processor -- start 2024-03-29T10:42:50.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.355+0000 7fa9ac28d640 1 -- start start 2024-03-29T10:42:50.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.355+0000 7fa9ac28d640 10 monclient: init 2024-03-29T10:42:50.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.356+0000 7fa9ac28d640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:50.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.356+0000 7fa9ac28d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:50.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.356+0000 7fa9ac28d640 10 monclient(hunting): picked mon.noname-b con 0x7fa9a41422b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:50.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.360+0000 7fa9ac28d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9a4142690 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:50.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.360+0000 7fa9aa82a640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa9a40b17d0 0x7fa9a40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54090/0 (socket says 172.21.15.114:54090) 2024-03-29T10:42:50.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.360+0000 7fa9aa82a640 1 -- 172.21.15.114:0/3933209193 learned_addr learned my addr 172.21.15.114:0/3933209193 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:50.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.360+0000 7fa9ac28d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9a40b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:50.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.360+0000 7fa9ac28d640 1 -- 172.21.15.114:0/3933209193 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a4142c80 con 0x7fa9a40b17d0 2024-03-29T10:42:50.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.360+0000 7fa9ac28d640 1 -- 172.21.15.114:0/3933209193 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a40aae40 con 0x7fa9a4142690 2024-03-29T10:42:50.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.362+0000 7fa9ac28d640 1 -- 172.21.15.114:0/3933209193 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a413fbd0 con 0x7fa9a41422b0 2024-03-29T10:42:50.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.362+0000 7fa9ac28d640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:50.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.362+0000 7fa9a9027640 1 -- 172.21.15.114:0/3933209193 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200513485 0 0) 0x7fa9a4142c80 con 0x7fa9a40b17d0 2024-03-29T10:42:50.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.362+0000 7fa9ac28d640 10 monclient(hunting): authenticate will time out at 1035.602306s 2024-03-29T10:42:50.349 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient(hunting): _negotiate 2024-03-29T10:42:50.349 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:50.349 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:50.349 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:42:50.349 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 1 -- 172.21.15.114:0/3933209193 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa990003520 con 0x7fa9a40b17d0 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 1 -- 172.21.15.114:0/3933209193 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692971914 0 0) 0x7fa9a413fbd0 con 0x7fa9a41422b0 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient(hunting): _negotiate 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 1 -- 172.21.15.114:0/3933209193 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a4142c80 con 0x7fa9a41422b0 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 1 -- 172.21.15.114:0/3933209193 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3059629209 0 0) 0x7fa9a4142c80 con 0x7fa9a41422b0 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 1 -- 172.21.15.114:0/3933209193 >> v1:172.21.15.176:6789/0 conn(0x7fa9a4142690 legacy=0x7fa9a40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 1 -- 172.21.15.114:0/3933209193 >> v1:172.21.15.114:6789/0 conn(0x7fa9a40b17d0 legacy=0x7fa9a40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient: _finish_hunting 0 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 1 monclient: found mon.noname-b 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:42:50.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 1 -- 172.21.15.114:0/3933209193 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a4150930 con 0x7fa9a41422b0 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient: _finish_auth 0 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient: _check_auth_tickets 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 1 -- 172.21.15.114:0/3933209193 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa994001670 con 0x7fa9a41422b0 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9a9027640 10 monclient: dump: 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:50.351 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9ac28d640 5 monclient: authenticate success, global_id 4376 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.363+0000 7fa9ac28d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.364+0000 7fa9a9027640 1 -- 172.21.15.114:0/3933209193 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9940017d0 con 0x7fa9a41422b0 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.364+0000 7fa9a9027640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.364+0000 7fa9a9027640 1 -- 172.21.15.114:0/3933209193 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa994001b60 con 0x7fa9a41422b0 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.364+0000 7fa9ac28d640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.364+0000 7fa9ac28d640 4 set_mon_vals no callback set 2024-03-29T10:42:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.365+0000 7fa9ac28d640 10 monclient: shutdown 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.365+0000 7fa9a9027640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.365+0000 7fa9a9027640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.366+0000 7fa9a9027640 10 monclient: dump: 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.366+0000 7fa9ac28d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:50.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.366+0000 7fa9ac28d640 1 -- 172.21.15.114:0/3933209193 >> v1:172.21.15.114:6790/0 conn(0x7fa9a41422b0 legacy=0x7fa9a413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:50.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.366+0000 7fa9ac28d640 1 -- 172.21.15.114:0/3933209193 shutdown_connections 2024-03-29T10:42:50.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.366+0000 7fa9ac28d640 1 -- 172.21.15.114:0/3933209193 >> 172.21.15.114:0/3933209193 conn(0x7fa9a4066a00 msgr2=0x7fa9a4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:50.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.366+0000 7fa9ac28d640 1 -- 172.21.15.114:0/3933209193 shutdown_connections 2024-03-29T10:42:50.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.366+0000 7fa9ac28d640 1 -- 172.21.15.114:0/3933209193 wait complete. 2024-03-29T10:42:50.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.367+0000 7fa9ac28d640 10 monclient: build_initial_monmap 2024-03-29T10:42:50.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.367+0000 7fa9ac28d640 10 monclient: monmap: 2024-03-29T10:42:50.363 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:50.363 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:50.363 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:42:50.363 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:42:50.363 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:50.363 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:50.363 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:50.363 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:42:50.363 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:42:50.364 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.368+0000 7fa9ac28d640 1 librados: starting msgr at 2024-03-29T10:42:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.368+0000 7fa9ac28d640 1 librados: starting objecter 2024-03-29T10:42:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.368+0000 7fa9ac28d640 1 Processor -- start 2024-03-29T10:42:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.368+0000 7fa9ac28d640 1 -- start start 2024-03-29T10:42:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.368+0000 7fa9ac28d640 1 librados: setting wanted keys 2024-03-29T10:42:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.368+0000 7fa9ac28d640 1 librados: calling monclient init 2024-03-29T10:42:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.368+0000 7fa9ac28d640 10 monclient: init 2024-03-29T10:42:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.369+0000 7fa9ac28d640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.369+0000 7fa9ac28d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.369+0000 7fa9ac28d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9a40b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.370+0000 7fa9ac28d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9a41422b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.372+0000 7fa9aa029640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa9a40b17d0 0x7fa9a41316c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:57354/0 (socket says 172.21.15.114:57354) 2024-03-29T10:42:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.372+0000 7fa9a9828640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fa9a41422b0 0x7fa9a412fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:51493/0 (socket says 172.21.15.114:51493) 2024-03-29T10:42:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.373+0000 7fa9ac28d640 10 monclient(hunting): picked mon.noname-b con 0x7fa9a4142690 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.373+0000 7fa9ac28d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a40abd50 con 0x7fa9a40b17d0 2024-03-29T10:42:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.373+0000 7fa9ac28d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a40ac230 con 0x7fa9a4142690 2024-03-29T10:42:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.373+0000 7fa9ac28d640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a41337b0 con 0x7fa9a41422b0 2024-03-29T10:42:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.373+0000 7fa9ac28d640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.373+0000 7fa9aa82a640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fa9a4142690 0x7fa9a40ac700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:60913/0 (socket says 172.21.15.114:60913) 2024-03-29T10:42:50.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.373+0000 7fa9aa82a640 1 -- 172.21.15.114:0/406705162 learned_addr learned my addr 172.21.15.114:0/406705162 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:50.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.373+0000 7fa9ac28d640 10 monclient(hunting): authenticate will time out at 1035.613946s 2024-03-29T10:42:50.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 client.?.objecter ms_handle_connect 0x7fa9a4142690 2024-03-29T10:42:50.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:50.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 client.?.objecter ms_handle_connect 0x7fa9a41422b0 2024-03-29T10:42:50.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:50.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288698290 0 0) 0x7fa9a40ac230 con 0x7fa9a4142690 2024-03-29T10:42:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:42:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa988004eb0 con 0x7fa9a4142690 2024-03-29T10:42:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486847184 0 0) 0x7fa9a41337b0 con 0x7fa9a41422b0 2024-03-29T10:42:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.374+0000 7fa99affd640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a40ac230 con 0x7fa9a41422b0 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 741180759 0 0) 0x7fa988004eb0 con 0x7fa9a4142690 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9a41337b0 con 0x7fa9a4142690 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa9a0001950 con 0x7fa9a4142690 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 10 monclient(hunting): dump: 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:50.368 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1962538105 0 0) 0x7fa9a40ac230 con 0x7fa9a41422b0 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa988004eb0 con 0x7fa9a41422b0 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa99c001b10 con 0x7fa9a41422b0 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.375+0000 7fa99affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 10 monclient(hunting): dump: 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:50.369 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2274147659 0 0) 0x7fa988004eb0 con 0x7fa9a41422b0 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 >> v1:172.21.15.176:6789/0 conn(0x7fa9a4142690 legacy=0x7fa9a40ac700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 >> v1:172.21.15.114:6789/0 conn(0x7fa9a40b17d0 legacy=0x7fa9a41316c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 1 monclient: found mon.c 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:50.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a4131dd0 con 0x7fa9a41422b0 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 10 monclient: _finish_auth 0 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa99affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa9ac28d640 5 monclient: authenticate success, global_id 4379 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa9ac28d640 10 monclient: _renew_subs 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa9ac28d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa9ac28d640 1 -- 172.21.15.114:0/406705162 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a4132040 con 0x7fa9a41422b0 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa9ac28d640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa9ac28d640 10 monclient: _renew_subs 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa9ac28d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.376+0000 7fa9ac28d640 1 -- 172.21.15.114:0/406705162 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a41304b0 con 0x7fa9a41422b0 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.377+0000 7fa9ac28d640 1 librados: init done 2024-03-29T10:42:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.378+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa99c001c70 con 0x7fa9a41422b0 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.378+0000 7fa99affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.378+0000 7fa99bfff640 4 set_mon_vals no callback set 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.379+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa99c003b30 con 0x7fa9a41422b0 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.379+0000 7fa99affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.379+0000 7fa99affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.379+0000 7fa99affd640 10 monclient: dump: 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:50.372 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:50.373 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:50.373 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:50.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.380+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fa99c003c90 con 0x7fa9a41422b0 2024-03-29T10:42:50.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.381+0000 7fa998ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:50.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.381+0000 7fa998ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:50.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.381+0000 7fa998ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:42:50.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.381+0000 7fa998ff9640 1 -- 172.21.15.114:0/406705162 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98c005080 con 0x7fa9a41422b0 2024-03-29T10:42:50.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.381+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fa99c084ca0 con 0x7fa9a41422b0 2024-03-29T10:42:50.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.381+0000 7fa99affd640 10 client.4379.objecter ms_dispatch 0x7fa9a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:50.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.382+0000 7fa99affd640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:50.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.382+0000 7fa99affd640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:50.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.382+0000 7fa99affd640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T10:42:50.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.382+0000 7fa9aa029640 10 monclient: get_auth_request con 0x7fa98806de60 auth_method 0 2024-03-29T10:42:50.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.383+0000 7fa99affd640 10 client.4379.objecter ms_handle_connect 0x7fa98806de60 2024-03-29T10:42:50.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.389+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa99c050580 con 0x7fa9a41422b0 2024-03-29T10:42:50.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.389+0000 7fa99affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:50.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.389+0000 7fa99affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:50.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.717+0000 7fa998ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:50.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.717+0000 7fa998ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:50.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.717+0000 7fa998ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:42:50.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.717+0000 7fa998ff9640 1 -- 172.21.15.114:0/406705162 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa98c005330 con 0x7fa9a41422b0 2024-03-29T10:42:50.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.718+0000 7fa99affd640 1 -- 172.21.15.114:0/406705162 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fa99c0524b0 con 0x7fa9a41422b0 2024-03-29T10:42:50.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.718+0000 7fa99affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:50.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.718+0000 7fa99affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:50.702 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:42:50.703 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:50.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.723+0000 7fa998ff9640 10 librados: watch_flush enter 2024-03-29T10:42:50.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.723+0000 7fa998ff9640 10 librados: watch_flush exit 2024-03-29T10:42:50.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.723+0000 7fa998ff9640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T10:42:50.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.723+0000 7fa998ff9640 10 client.4379.objecter successfully canceled tick 2024-03-29T10:42:50.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.724+0000 7fa998ff9640 1 -- 172.21.15.114:0/406705162 >> v1:172.21.15.114:6816/775044307 conn(0x7fa98806de60 legacy=0x7fa988070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:50.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.724+0000 7fa998ff9640 10 monclient: shutdown 2024-03-29T10:42:50.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.724+0000 7fa998ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:50.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.724+0000 7fa998ff9640 1 -- 172.21.15.114:0/406705162 >> v1:172.21.15.114:6790/0 conn(0x7fa9a41422b0 legacy=0x7fa9a412fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:50.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.725+0000 7fa998ff9640 1 -- 172.21.15.114:0/406705162 shutdown_connections 2024-03-29T10:42:50.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.725+0000 7fa998ff9640 1 -- 172.21.15.114:0/406705162 >> 172.21.15.114:0/406705162 conn(0x7fa9a4066a00 msgr2=0x7fa9a414f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:50.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.725+0000 7fa998ff9640 1 -- 172.21.15.114:0/406705162 shutdown_connections 2024-03-29T10:42:50.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.725+0000 7fa998ff9640 1 -- 172.21.15.114:0/406705162 wait complete. 2024-03-29T10:42:50.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:50.725+0000 7fa998ff9640 1 librados: shutdown 2024-03-29T10:42:50.724 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:57.532 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.700+0000 7f399f682640 10 monclient: get_monmap_and_config 2024-03-29T10:42:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.700+0000 7f399f682640 10 monclient: build_initial_monmap 2024-03-29T10:42:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.700+0000 7f399f682640 10 monclient: monmap: 2024-03-29T10:42:57.684 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:57.685 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:57.685 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:42:57.701848+0000 2024-03-29T10:42:57.685 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:42:57.701848+0000 2024-03-29T10:42:57.685 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:57.685 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:57.685 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:57.685 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:42:57.685 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:42:57.685 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:57.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.701+0000 7f399f682640 1 Processor -- start 2024-03-29T10:42:57.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.702+0000 7f399f682640 1 -- start start 2024-03-29T10:42:57.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.702+0000 7f399f682640 10 monclient: init 2024-03-29T10:42:57.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.703+0000 7f399f682640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:57.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.703+0000 7f399f682640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:57.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.703+0000 7f399f682640 10 monclient(hunting): picked mon.noname-b con 0x7f39981457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:57.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.703+0000 7f399f682640 10 monclient(hunting): picked mon.noname-a con 0x7f399813f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:57.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.703+0000 7f399f682640 10 monclient(hunting): picked mon.noname-c con 0x7f39980a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:57.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.703+0000 7f399f682640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39980aaef0 con 0x7f399813f650 2024-03-29T10:42:57.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.703+0000 7f399f682640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3998145bd0 con 0x7f39980a9d70 2024-03-29T10:42:57.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.703+0000 7f399f682640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3998145eb0 con 0x7f39981457f0 2024-03-29T10:42:57.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.703+0000 7f399f682640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.703+0000 7f399f682640 10 monclient(hunting): authenticate will time out at 1042.944257s 2024-03-29T10:42:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.705+0000 7f399d41e640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f39981457f0 0x7f399813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:57680/0 (socket says 172.21.15.114:57680) 2024-03-29T10:42:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.705+0000 7f399d41e640 1 -- 172.21.15.114:0/2925055018 learned_addr learned my addr 172.21.15.114:0/2925055018 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 1 -- 172.21.15.114:0/2925055018 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063566817 0 0) 0x7f3998145eb0 con 0x7f39981457f0 2024-03-29T10:42:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:42:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 1 -- 172.21.15.114:0/2925055018 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3984003520 con 0x7f39981457f0 2024-03-29T10:42:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 1 -- 172.21.15.114:0/2925055018 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890790602 0 0) 0x7f39980aaef0 con 0x7f399813f650 2024-03-29T10:42:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:57.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 10 monclient(hunting): my global_id is 4374 2024-03-29T10:42:57.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 1 -- 172.21.15.114:0/2925055018 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3998145eb0 con 0x7f399813f650 2024-03-29T10:42:57.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 1 -- 172.21.15.114:0/2925055018 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 683677106 0 0) 0x7f3998145eb0 con 0x7f399813f650 2024-03-29T10:42:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 1 -- 172.21.15.114:0/2925055018 >> v1:172.21.15.114:6790/0 conn(0x7f39981457f0 legacy=0x7f399813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 1 -- 172.21.15.114:0/2925055018 >> v1:172.21.15.176:6789/0 conn(0x7f39980a9d70 legacy=0x7f399814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 1 monclient: found mon.noname-a 2024-03-29T10:42:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:42:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.706+0000 7f398ffff640 1 -- 172.21.15.114:0/2925055018 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3998146340 con 0x7f399813f650 2024-03-29T10:42:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 10 monclient: _finish_auth 0 2024-03-29T10:42:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f399f682640 5 monclient: authenticate success, global_id 4374 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f399f682640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 1 -- 172.21.15.114:0/2925055018 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3994001670 con 0x7f399813f650 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 10 monclient: dump: 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:57.693 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f399f682640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 1 -- 172.21.15.114:0/2925055018 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39940017d0 con 0x7f399813f650 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f399f682640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f399f682640 4 set_mon_vals no callback set 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 1 -- 172.21.15.114:0/2925055018 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3994001c20 con 0x7f399813f650 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.707+0000 7f398ffff640 10 monclient: dump: 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:57.694 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.708+0000 7f399f682640 10 monclient: shutdown 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.708+0000 7f399f682640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.708+0000 7f399f682640 1 -- 172.21.15.114:0/2925055018 >> v1:172.21.15.114:6789/0 conn(0x7f399813f650 legacy=0x7f39980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.708+0000 7f399f682640 1 -- 172.21.15.114:0/2925055018 shutdown_connections 2024-03-29T10:42:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.708+0000 7f399f682640 1 -- 172.21.15.114:0/2925055018 >> 172.21.15.114:0/2925055018 conn(0x7f3998066a00 msgr2=0x7f3998066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:57.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.709+0000 7f399f682640 1 -- 172.21.15.114:0/2925055018 shutdown_connections 2024-03-29T10:42:57.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.709+0000 7f399f682640 1 -- 172.21.15.114:0/2925055018 wait complete. 2024-03-29T10:42:57.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.710+0000 7f399f682640 10 monclient: build_initial_monmap 2024-03-29T10:42:57.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.710+0000 7f399f682640 10 monclient: monmap: 2024-03-29T10:42:57.696 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:42:57.696 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.711+0000 7f399f682640 1 librados: starting msgr at 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.711+0000 7f399f682640 1 librados: starting objecter 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.711+0000 7f399f682640 1 Processor -- start 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.711+0000 7f399f682640 1 -- start start 2024-03-29T10:42:57.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.711+0000 7f399f682640 1 librados: setting wanted keys 2024-03-29T10:42:57.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.711+0000 7f399f682640 1 librados: calling monclient init 2024-03-29T10:42:57.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.711+0000 7f399f682640 10 monclient: init 2024-03-29T10:42:57.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.712+0000 7f399f682640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:57.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.712+0000 7f399f682640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:57.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.712+0000 7f399f682640 10 monclient(hunting): picked mon.noname-a con 0x7f39980a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:42:57.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.712+0000 7f399f682640 10 monclient(hunting): picked mon.noname-c con 0x7f399813f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:42:57.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.712+0000 7f399f682640 10 monclient(hunting): picked mon.noname-b con 0x7f39981457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:42:57.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.712+0000 7f399f682640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3998131730 con 0x7f39980a9d70 2024-03-29T10:42:57.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.712+0000 7f399f682640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399813cb00 con 0x7f39981457f0 2024-03-29T10:42:57.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.712+0000 7f399f682640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399813e920 con 0x7f399813f650 2024-03-29T10:42:57.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.712+0000 7f399f682640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.712+0000 7f399f682640 10 monclient(hunting): authenticate will time out at 1042.953201s 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.714+0000 7f399dc1f640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f39981457f0 0x7f3998131020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:54967/0 (socket says 172.21.15.114:54967) 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.714+0000 7f399dc1f640 1 -- 172.21.15.114:0/5218755 learned_addr learned my addr 172.21.15.114:0/5218755 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 10 client.?.objecter ms_handle_connect 0x7f39981457f0 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617140989 0 0) 0x7f399813cb00 con 0x7f39981457f0 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c004eb0 con 0x7f39981457f0 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2184569540 0 0) 0x7f397c004eb0 con 0x7f39981457f0 2024-03-29T10:42:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f399813cb00 con 0x7f39981457f0 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3990001d30 con 0x7f39981457f0 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 10 monclient(hunting): dump: 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:57.702 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:57.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.715+0000 7f398dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:57.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.716+0000 7f398dffb640 10 client.?.objecter ms_handle_connect 0x7f399813f650 2024-03-29T10:42:57.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.716+0000 7f398dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:57.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.716+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074887303 0 0) 0x7f399813e920 con 0x7f399813f650 2024-03-29T10:42:57.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.716+0000 7f398dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:57.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.716+0000 7f398dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.716+0000 7f398dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.716+0000 7f398dffb640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:42:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.716+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c004eb0 con 0x7f399813f650 2024-03-29T10:42:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.717+0000 7f398dffb640 10 client.?.objecter ms_handle_connect 0x7f39980a9d70 2024-03-29T10:42:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.717+0000 7f398dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.717+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438162510 0 0) 0x7f3998131730 con 0x7f39980a9d70 2024-03-29T10:42:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.717+0000 7f398dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.717+0000 7f398dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.717+0000 7f398dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.717+0000 7f398dffb640 10 monclient(hunting): my global_id is 4377 2024-03-29T10:42:57.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.717+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f399813e920 con 0x7f39980a9d70 2024-03-29T10:42:57.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3551431883 0 0) 0x7f399813cb00 con 0x7f39981457f0 2024-03-29T10:42:57.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 >> v1:172.21.15.114:6790/0 conn(0x7f399813f650 legacy=0x7f3998130910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 >> v1:172.21.15.114:6789/0 conn(0x7f39980a9d70 legacy=0x7f399813d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:57.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 1 monclient: found mon.b 2024-03-29T10:42:57.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:57.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:57.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399813d9e0 con 0x7f39981457f0 2024-03-29T10:42:57.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3990002030 con 0x7f39981457f0 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3990003de0 con 0x7f39981457f0 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398dffb640 10 monclient: dump: 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:57.708 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:42:57.709 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:57.709 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:42:57.709 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:42:57.709 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:42:57.709 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:42:57.709 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:42:57.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.718+0000 7f398f7fe640 4 set_mon_vals no callback set 2024-03-29T10:42:57.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.719+0000 7f399f682640 5 monclient: authenticate success, global_id 4348 2024-03-29T10:42:57.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.719+0000 7f399f682640 10 monclient: _renew_subs 2024-03-29T10:42:57.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.719+0000 7f399f682640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:57.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.719+0000 7f399f682640 1 -- 172.21.15.114:0/5218755 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3998136a60 con 0x7f39981457f0 2024-03-29T10:42:57.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.719+0000 7f399f682640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:57.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.719+0000 7f399f682640 10 monclient: _renew_subs 2024-03-29T10:42:57.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.719+0000 7f399f682640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:57.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.719+0000 7f399f682640 1 -- 172.21.15.114:0/5218755 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3998139ed0 con 0x7f39981457f0 2024-03-29T10:42:57.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.719+0000 7f399f682640 1 librados: init done 2024-03-29T10:42:57.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.721+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f3990003f40 con 0x7f39981457f0 2024-03-29T10:42:57.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.721+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f3990084f10 con 0x7f39981457f0 2024-03-29T10:42:57.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.722+0000 7f399f682640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:57.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.722+0000 7f399f682640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:57.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.722+0000 7f399f682640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:57.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.722+0000 7f399f682640 1 -- 172.21.15.114:0/5218755 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3998136bf0 con 0x7f39981457f0 2024-03-29T10:42:57.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.722+0000 7f399d41e640 10 monclient: get_auth_request con 0x7f397c06de50 auth_method 0 2024-03-29T10:42:57.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.721+0000 7f398dffb640 10 client.4348.objecter ms_dispatch 0x7f3998000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:57.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.722+0000 7f398dffb640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:57.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.722+0000 7f398dffb640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:57.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.722+0000 7f398dffb640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T10:42:57.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.723+0000 7f398dffb640 10 client.4348.objecter ms_handle_connect 0x7f397c06de50 2024-03-29T10:42:57.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.731+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39900507f0 con 0x7f39981457f0 2024-03-29T10:42:57.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.731+0000 7f398dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:57.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:57.731+0000 7f398dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:58.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.054+0000 7f399f682640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:58.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.054+0000 7f399f682640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:58.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.054+0000 7f399f682640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:42:58.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.054+0000 7f399f682640 1 -- 172.21.15.114:0/5218755 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f399813cc90 con 0x7f39981457f0 2024-03-29T10:42:58.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.056+0000 7f398dffb640 1 -- 172.21.15.114:0/5218755 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f3990052720 con 0x7f39981457f0 2024-03-29T10:42:58.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.056+0000 7f398dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:58.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.056+0000 7f398dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:58.041 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:42:58.041 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.060+0000 7f399f682640 10 librados: watch_flush enter 2024-03-29T10:42:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.060+0000 7f399f682640 10 librados: watch_flush exit 2024-03-29T10:42:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.060+0000 7f399f682640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T10:42:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.060+0000 7f399f682640 10 client.4348.objecter successfully canceled tick 2024-03-29T10:42:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.060+0000 7f399f682640 1 -- 172.21.15.114:0/5218755 >> v1:172.21.15.114:6816/775044307 conn(0x7f397c06de50 legacy=0x7f397c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.060+0000 7f399f682640 10 monclient: shutdown 2024-03-29T10:42:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.060+0000 7f399f682640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.060+0000 7f399f682640 1 -- 172.21.15.114:0/5218755 >> v1:172.21.15.176:6789/0 conn(0x7f39981457f0 legacy=0x7f3998131020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:58.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.060+0000 7f399f682640 1 -- 172.21.15.114:0/5218755 shutdown_connections 2024-03-29T10:42:58.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.060+0000 7f399f682640 1 -- 172.21.15.114:0/5218755 >> 172.21.15.114:0/5218755 conn(0x7f3998066a00 msgr2=0x7f399813b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:58.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.061+0000 7f399f682640 1 -- 172.21.15.114:0/5218755 shutdown_connections 2024-03-29T10:42:58.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.061+0000 7f399f682640 1 -- 172.21.15.114:0/5218755 wait complete. 2024-03-29T10:42:58.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:42:58.061+0000 7f399f682640 1 librados: shutdown 2024-03-29T10:42:58.062 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:04.870 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:05.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.032+0000 7fd24b6e6640 10 monclient: get_monmap_and_config 2024-03-29T10:43:05.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.032+0000 7fd24b6e6640 10 monclient: build_initial_monmap 2024-03-29T10:43:05.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.032+0000 7fd24b6e6640 10 monclient: monmap: 2024-03-29T10:43:05.016 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:05.016 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:05.017 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:43:05.034130+0000 2024-03-29T10:43:05.017 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:43:05.034130+0000 2024-03-29T10:43:05.017 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:05.017 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:05.017 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:05.017 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:43:05.017 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:43:05.017 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:05.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.034+0000 7fd24b6e6640 1 Processor -- start 2024-03-29T10:43:05.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.034+0000 7fd24b6e6640 1 -- start start 2024-03-29T10:43:05.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.034+0000 7fd24b6e6640 10 monclient: init 2024-03-29T10:43:05.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.035+0000 7fd24b6e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:05.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.035+0000 7fd24b6e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:05.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.035+0000 7fd24b6e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd2440b0860 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:05.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.035+0000 7fd24b6e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd24413ef30 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:05.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.036+0000 7fd24b6e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd2440af8e0 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:05.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.036+0000 7fd24b6e6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2440b0c40 con 0x7fd24413ef30 2024-03-29T10:43:05.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.036+0000 7fd24b6e6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2440aed10 con 0x7fd2440af8e0 2024-03-29T10:43:05.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.036+0000 7fd24b6e6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2440afd50 con 0x7fd2440b0860 2024-03-29T10:43:05.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.036+0000 7fd24b6e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:05.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.036+0000 7fd24b6e6640 10 monclient(hunting): authenticate will time out at 1050.277800s 2024-03-29T10:43:05.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.037+0000 7fd249482640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fd2440b0860 0x7fd24413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:57436/0 (socket says 172.21.15.114:57436) 2024-03-29T10:43:05.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.037+0000 7fd249482640 1 -- 172.21.15.114:0/2673212996 learned_addr learned my addr 172.21.15.114:0/2673212996 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.039+0000 7fd23bfff640 1 -- 172.21.15.114:0/2673212996 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135708420 0 0) 0x7fd2440afd50 con 0x7fd2440b0860 2024-03-29T10:43:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:43:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 1 -- 172.21.15.114:0/2673212996 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd230003520 con 0x7fd2440b0860 2024-03-29T10:43:05.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 1 -- 172.21.15.114:0/2673212996 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 396615975 0 0) 0x7fd230003520 con 0x7fd2440b0860 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 1 -- 172.21.15.114:0/2673212996 >> v1:172.21.15.176:6789/0 conn(0x7fd2440af8e0 legacy=0x7fd244149a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 1 -- 172.21.15.114:0/2673212996 >> v1:172.21.15.114:6789/0 conn(0x7fd24413ef30 legacy=0x7fd2440af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 1 monclient: found mon.noname-b 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 1 -- 172.21.15.114:0/2673212996 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2440ab1b0 con 0x7fd2440b0860 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 10 monclient: _finish_auth 0 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 1 -- 172.21.15.114:0/2673212996 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd234001690 con 0x7fd2440b0860 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.040+0000 7fd23bfff640 10 monclient: dump: 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.043+0000 7fd24b6e6640 5 monclient: authenticate success, global_id 4388 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.043+0000 7fd24b6e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:05.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.044+0000 7fd23bfff640 1 -- 172.21.15.114:0/2673212996 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2340018f0 con 0x7fd2440b0860 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.044+0000 7fd23bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.044+0000 7fd23bfff640 1 -- 172.21.15.114:0/2673212996 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd234001690 con 0x7fd2440b0860 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.044+0000 7fd23bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.044+0000 7fd23bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.044+0000 7fd23bfff640 10 monclient: dump: 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:05.031 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:05.032 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:05.032 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:05.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.044+0000 7fd24b6e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:05.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.044+0000 7fd24b6e6640 4 set_mon_vals no callback set 2024-03-29T10:43:05.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.045+0000 7fd24b6e6640 10 monclient: shutdown 2024-03-29T10:43:05.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.045+0000 7fd24b6e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:05.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.045+0000 7fd24b6e6640 1 -- 172.21.15.114:0/2673212996 >> v1:172.21.15.114:6790/0 conn(0x7fd2440b0860 legacy=0x7fd24413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:05.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.045+0000 7fd24b6e6640 1 -- 172.21.15.114:0/2673212996 shutdown_connections 2024-03-29T10:43:05.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.045+0000 7fd24b6e6640 1 -- 172.21.15.114:0/2673212996 >> 172.21.15.114:0/2673212996 conn(0x7fd24406b660 msgr2=0x7fd24406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.045+0000 7fd24b6e6640 1 -- 172.21.15.114:0/2673212996 shutdown_connections 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.046+0000 7fd24b6e6640 1 -- 172.21.15.114:0/2673212996 wait complete. 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.047+0000 7fd24b6e6640 10 monclient: build_initial_monmap 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.047+0000 7fd24b6e6640 10 monclient: monmap: 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:05.035 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.048+0000 7fd24b6e6640 1 librados: starting msgr at 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.048+0000 7fd24b6e6640 1 librados: starting objecter 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.048+0000 7fd24b6e6640 1 Processor -- start 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.048+0000 7fd24b6e6640 1 -- start start 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.048+0000 7fd24b6e6640 1 librados: setting wanted keys 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.048+0000 7fd24b6e6640 1 librados: calling monclient init 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.048+0000 7fd24b6e6640 10 monclient: init 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.049+0000 7fd24b6e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.049+0000 7fd24b6e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.049+0000 7fd24b6e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd2440af8e0 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.049+0000 7fd24b6e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd2440b0860 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.049+0000 7fd24b6e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd24413ef30 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.049+0000 7fd24b6e6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24406fdd0 con 0x7fd2440af8e0 2024-03-29T10:43:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.049+0000 7fd24b6e6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24406f940 con 0x7fd24413ef30 2024-03-29T10:43:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.049+0000 7fd24b6e6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd244070d00 con 0x7fd2440b0860 2024-03-29T10:43:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.049+0000 7fd24b6e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.049+0000 7fd24b6e6640 10 monclient(hunting): authenticate will time out at 1050.290978s 2024-03-29T10:43:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.050+0000 7fd248c81640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fd2440b0860 0x7fd244134790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33680/0 (socket says 172.21.15.114:33680) 2024-03-29T10:43:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.050+0000 7fd248c81640 1 -- 172.21.15.114:0/1727543731 learned_addr learned my addr 172.21.15.114:0/1727543731 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.053+0000 7fd239ffb640 10 client.?.objecter ms_handle_connect 0x7fd2440b0860 2024-03-29T10:43:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.053+0000 7fd239ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.053+0000 7fd239ffb640 10 client.?.objecter ms_handle_connect 0x7fd24413ef30 2024-03-29T10:43:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.053+0000 7fd239ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.053+0000 7fd239ffb640 10 client.?.objecter ms_handle_connect 0x7fd2440af8e0 2024-03-29T10:43:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.053+0000 7fd239ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991151677 0 0) 0x7fd244070d00 con 0x7fd2440b0860 2024-03-29T10:43:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd228004eb0 con 0x7fd2440b0860 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434497260 0 0) 0x7fd24406fdd0 con 0x7fd2440af8e0 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd244070d00 con 0x7fd2440af8e0 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2407400871 0 0) 0x7fd228004eb0 con 0x7fd2440b0860 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd24406fdd0 con 0x7fd2440b0860 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd23c0017d0 con 0x7fd2440b0860 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:05.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): dump: 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.054+0000 7fd239ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4286976664 0 0) 0x7fd24406fdd0 con 0x7fd2440b0860 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 >> v1:172.21.15.176:6789/0 conn(0x7fd24413ef30 legacy=0x7fd2441e71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 >> v1:172.21.15.114:6789/0 conn(0x7fd2440af8e0 legacy=0x7fd2441db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:43:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 1 monclient: found mon.c 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24406f420 con 0x7fd2440b0860 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 10 monclient: _finish_auth 0 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd23c001ce0 con 0x7fd2440b0860 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd23c0017d0 con 0x7fd2440b0860 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd239ffb640 10 monclient: dump: 2024-03-29T10:43:05.046 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd23affd640 4 set_mon_vals no callback set 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd24b6e6640 5 monclient: authenticate success, global_id 4391 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd24b6e6640 10 monclient: _renew_subs 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd24b6e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:05.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.055+0000 7fd24b6e6640 1 -- 172.21.15.114:0/1727543731 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24406e910 con 0x7fd2440b0860 2024-03-29T10:43:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.057+0000 7fd24b6e6640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.057+0000 7fd24b6e6640 10 monclient: _renew_subs 2024-03-29T10:43:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.057+0000 7fd24b6e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.057+0000 7fd24b6e6640 1 -- 172.21.15.114:0/1727543731 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd244071950 con 0x7fd2440b0860 2024-03-29T10:43:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.057+0000 7fd24b6e6640 1 librados: init done 2024-03-29T10:43:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.058+0000 7fd24b6e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.059+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fd23c01a750 con 0x7fd2440b0860 2024-03-29T10:43:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.059+0000 7fd24b6e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.059+0000 7fd24b6e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.059+0000 7fd24b6e6640 1 -- 172.21.15.114:0/1727543731 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd244075f10 con 0x7fd2440b0860 2024-03-29T10:43:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.060+0000 7fd249482640 10 monclient: get_auth_request con 0x7fd22806e070 auth_method 0 2024-03-29T10:43:05.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.060+0000 7fd239ffb640 10 client.4391.objecter ms_handle_connect 0x7fd22806e070 2024-03-29T10:43:05.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.063+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fd23c0015f0 con 0x7fd2440b0860 2024-03-29T10:43:05.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.063+0000 7fd239ffb640 10 client.4391.objecter ms_dispatch 0x7fd244000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:05.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.063+0000 7fd239ffb640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:05.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.063+0000 7fd239ffb640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:05.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.063+0000 7fd239ffb640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-29T10:43:05.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.068+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd23c04fcb0 con 0x7fd2440b0860 2024-03-29T10:43:05.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.068+0000 7fd239ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:05.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.068+0000 7fd239ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:05.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.391+0000 7fd24b6e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:05.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.391+0000 7fd24b6e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:05.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.391+0000 7fd24b6e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:05.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.391+0000 7fd24b6e6640 1 -- 172.21.15.114:0/1727543731 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd244072330 con 0x7fd2440b0860 2024-03-29T10:43:05.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.392+0000 7fd239ffb640 1 -- 172.21.15.114:0/1727543731 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fd23c051db0 con 0x7fd2440b0860 2024-03-29T10:43:05.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.392+0000 7fd239ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:05.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.392+0000 7fd239ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:05.377 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:43:05.377 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:05.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.395+0000 7fd24b6e6640 10 librados: watch_flush enter 2024-03-29T10:43:05.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.397+0000 7fd24b6e6640 10 librados: watch_flush exit 2024-03-29T10:43:05.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.397+0000 7fd24b6e6640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-29T10:43:05.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.397+0000 7fd24b6e6640 10 client.4391.objecter successfully canceled tick 2024-03-29T10:43:05.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.398+0000 7fd24b6e6640 1 -- 172.21.15.114:0/1727543731 >> v1:172.21.15.114:6816/775044307 conn(0x7fd22806e070 legacy=0x7fd228070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:05.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.398+0000 7fd24b6e6640 10 monclient: shutdown 2024-03-29T10:43:05.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.398+0000 7fd24b6e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:05.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.398+0000 7fd24b6e6640 1 -- 172.21.15.114:0/1727543731 >> v1:172.21.15.114:6790/0 conn(0x7fd2440b0860 legacy=0x7fd244134790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:05.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.398+0000 7fd24b6e6640 1 -- 172.21.15.114:0/1727543731 shutdown_connections 2024-03-29T10:43:05.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.398+0000 7fd24b6e6640 1 -- 172.21.15.114:0/1727543731 >> 172.21.15.114:0/1727543731 conn(0x7fd24406b660 msgr2=0x7fd2440ac070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:05.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.398+0000 7fd24b6e6640 1 -- 172.21.15.114:0/1727543731 shutdown_connections 2024-03-29T10:43:05.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.398+0000 7fd24b6e6640 1 -- 172.21.15.114:0/1727543731 wait complete. 2024-03-29T10:43:05.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:05.398+0000 7fd24b6e6640 1 librados: shutdown 2024-03-29T10:43:05.399 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:12.206 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:12.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.368+0000 7f859e619640 10 monclient: get_monmap_and_config 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.368+0000 7f859e619640 10 monclient: build_initial_monmap 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.368+0000 7f859e619640 10 monclient: monmap: 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:43:12.369770+0000 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:43:12.369770+0000 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:43:12.352 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:12.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.369+0000 7f859e619640 1 Processor -- start 2024-03-29T10:43:12.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.369+0000 7f859e619640 1 -- start start 2024-03-29T10:43:12.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.369+0000 7f859e619640 10 monclient: init 2024-03-29T10:43:12.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.371+0000 7f859e619640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:12.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.371+0000 7f859e619640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:12.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.371+0000 7f859e619640 10 monclient(hunting): picked mon.noname-b con 0x7f85900371c0 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:12.355 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.372+0000 7f859d617640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f85900371c0 0x7f85900c2870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:51462/0 (socket says 172.21.15.114:51462) 2024-03-29T10:43:12.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.373+0000 7f859e619640 10 monclient(hunting): picked mon.noname-c con 0x7f859002cec0 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:12.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.373+0000 7f859d617640 1 -- 172.21.15.114:0/3435588739 learned_addr learned my addr 172.21.15.114:0/3435588739 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:12.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f859e619640 10 monclient(hunting): picked mon.noname-a con 0x7f85900c2f80 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:12.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f859e619640 1 -- 172.21.15.114:0/3435588739 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859002d4a0 con 0x7f85900c2f80 2024-03-29T10:43:12.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f859e619640 1 -- 172.21.15.114:0/3435588739 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859002e360 con 0x7f859002cec0 2024-03-29T10:43:12.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f859e619640 1 -- 172.21.15.114:0/3435588739 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859002e640 con 0x7f85900371c0 2024-03-29T10:43:12.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f859e619640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:12.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f859e619640 10 monclient(hunting): authenticate will time out at 1057.617822s 2024-03-29T10:43:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f858ffff640 1 -- 172.21.15.114:0/3435588739 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414678740 0 0) 0x7f859002d4a0 con 0x7f85900c2f80 2024-03-29T10:43:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f858ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f858ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f858ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f858ffff640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:43:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f858ffff640 1 -- 172.21.15.114:0/3435588739 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8584003520 con 0x7f85900c2f80 2024-03-29T10:43:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f858ffff640 1 -- 172.21.15.114:0/3435588739 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788779240 0 0) 0x7f859002e640 con 0x7f85900371c0 2024-03-29T10:43:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f858ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f858ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.376+0000 7f858ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.377+0000 7f858ffff640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.377+0000 7f858ffff640 1 -- 172.21.15.114:0/3435588739 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f859002d4a0 con 0x7f85900371c0 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.377+0000 7f858ffff640 1 -- 172.21.15.114:0/3435588739 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1602659330 0 0) 0x7f859002d4a0 con 0x7f85900371c0 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.377+0000 7f858ffff640 1 -- 172.21.15.114:0/3435588739 >> v1:172.21.15.176:6789/0 conn(0x7f859002cec0 legacy=0x7f85900c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.377+0000 7f858ffff640 1 -- 172.21.15.114:0/3435588739 >> v1:172.21.15.114:6789/0 conn(0x7f85900c2f80 legacy=0x7f85900c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.377+0000 7f858ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.377+0000 7f858ffff640 1 monclient: found mon.noname-b 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 1 -- 172.21.15.114:0/3435588739 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85900342c0 con 0x7f85900371c0 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 10 monclient: _finish_auth 0 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:12.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 1 -- 172.21.15.114:0/3435588739 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8588001690 con 0x7f85900371c0 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 10 monclient: dump: 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:12.365 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 1 -- 172.21.15.114:0/3435588739 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8588001920 con 0x7f85900371c0 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.378+0000 7f858ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.379+0000 7f858ffff640 1 -- 172.21.15.114:0/3435588739 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8588001cb0 con 0x7f85900371c0 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.379+0000 7f858ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.379+0000 7f858ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.379+0000 7f858ffff640 10 monclient: dump: 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:12.366 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:12.367 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:12.367 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:12.367 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:12.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.379+0000 7f859e619640 5 monclient: authenticate success, global_id 4394 2024-03-29T10:43:12.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.379+0000 7f859e619640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:12.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.379+0000 7f859e619640 4 set_mon_vals no callback set 2024-03-29T10:43:12.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.381+0000 7f859e619640 10 monclient: shutdown 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.381+0000 7f859e619640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.381+0000 7f859e619640 1 -- 172.21.15.114:0/3435588739 >> v1:172.21.15.114:6790/0 conn(0x7f85900371c0 legacy=0x7f85900c2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.382+0000 7f859e619640 1 -- 172.21.15.114:0/3435588739 shutdown_connections 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.382+0000 7f859e619640 1 -- 172.21.15.114:0/3435588739 >> 172.21.15.114:0/3435588739 conn(0x7f859003a590 msgr2=0x7f85900367e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.382+0000 7f859e619640 1 -- 172.21.15.114:0/3435588739 shutdown_connections 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.382+0000 7f859e619640 1 -- 172.21.15.114:0/3435588739 wait complete. 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.383+0000 7f859e619640 10 monclient: build_initial_monmap 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.383+0000 7f859e619640 10 monclient: monmap: 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:43:12.369 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.384+0000 7f859e619640 1 librados: starting msgr at 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.384+0000 7f859e619640 1 librados: starting objecter 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.384+0000 7f859e619640 1 Processor -- start 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.385+0000 7f859e619640 1 -- start start 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.385+0000 7f859e619640 1 librados: setting wanted keys 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.385+0000 7f859e619640 1 librados: calling monclient init 2024-03-29T10:43:12.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.385+0000 7f859e619640 10 monclient: init 2024-03-29T10:43:12.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.386+0000 7f859e619640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:12.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.386+0000 7f859e619640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:12.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.386+0000 7f859e619640 10 monclient(hunting): picked mon.noname-c con 0x7f859002cec0 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:12.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.386+0000 7f859e619640 10 monclient(hunting): picked mon.noname-a con 0x7f85900371c0 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:12.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.386+0000 7f859e619640 10 monclient(hunting): picked mon.noname-b con 0x7f85900c2f80 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.389+0000 7f859e619640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85900c8fe0 con 0x7f85900371c0 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.389+0000 7f859e619640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85900c9270 con 0x7f85900c2f80 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f859e619640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8590036060 con 0x7f859002cec0 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f859e619640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f859ce16640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f85900371c0 0x7f85900bbe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49641/0 (socket says 172.21.15.114:49641) 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f859ce16640 1 -- 172.21.15.114:0/3504730724 learned_addr learned my addr 172.21.15.114:0/3504730724 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f859e619640 10 monclient(hunting): authenticate will time out at 1057.631105s 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 client.?.objecter ms_handle_connect 0x7f85900371c0 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 client.?.objecter ms_handle_connect 0x7f85900c2f80 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 client.?.objecter ms_handle_connect 0x7f859002cec0 2024-03-29T10:43:12.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402457264 0 0) 0x7f85900c8fe0 con 0x7f85900371c0 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f857c004eb0 con 0x7f85900371c0 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882650487 0 0) 0x7f8590036060 con 0x7f859002cec0 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.390+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85900c8fe0 con 0x7f859002cec0 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2114983718 0 0) 0x7f85900c8fe0 con 0x7f859002cec0 2024-03-29T10:43:12.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8590036060 con 0x7f859002cec0 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8588001550 con 0x7f859002cec0 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient(hunting): dump: 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:12.378 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 796696710 0 0) 0x7f857c004eb0 con 0x7f85900371c0 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85900c8fe0 con 0x7f85900371c0 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8594001910 con 0x7f85900371c0 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient(hunting): dump: 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:12.379 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 104097390 0 0) 0x7f8590036060 con 0x7f859002cec0 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 >> v1:172.21.15.176:6789/0 conn(0x7f85900c2f80 legacy=0x7f859015ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 >> v1:172.21.15.114:6789/0 conn(0x7f85900371c0 legacy=0x7f85900bbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 monclient: found mon.c 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85900c82a0 con 0x7f859002cec0 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient: _finish_auth 0 2024-03-29T10:43:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:43:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f859e619640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:43:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f859e619640 10 monclient: _renew_subs 2024-03-29T10:43:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f859e619640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85880016b0 con 0x7f859002cec0 2024-03-29T10:43:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f859e619640 1 -- 172.21.15.114:0/3504730724 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85900c84f0 con 0x7f859002cec0 2024-03-29T10:43:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.391+0000 7f858f7fe640 4 set_mon_vals no callback set 2024-03-29T10:43:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.392+0000 7f859e619640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.392+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8588004d90 con 0x7f859002cec0 2024-03-29T10:43:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.392+0000 7f858dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.392+0000 7f858dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.392+0000 7f858dffb640 10 monclient: dump: 2024-03-29T10:43:12.383 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:12.383 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.392+0000 7f859e619640 10 monclient: _renew_subs 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.392+0000 7f859e619640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.392+0000 7f859e619640 1 -- 172.21.15.114:0/3504730724 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85900bc780 con 0x7f859002cec0 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.392+0000 7f859e619640 1 librados: init done 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.394+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f858801ad60 con 0x7f859002cec0 2024-03-29T10:43:12.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.394+0000 7f857b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.394+0000 7f857b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.394+0000 7f857b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.394+0000 7f857b7fe640 1 -- 172.21.15.114:0/3504730724 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8590005120 con 0x7f859002cec0 2024-03-29T10:43:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.394+0000 7f859ce16640 10 monclient: get_auth_request con 0x7f857c06de80 auth_method 0 2024-03-29T10:43:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.395+0000 7f858dffb640 10 client.4397.objecter ms_handle_connect 0x7f857c06de80 2024-03-29T10:43:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.395+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f8588085ae0 con 0x7f859002cec0 2024-03-29T10:43:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.395+0000 7f858dffb640 10 client.4397.objecter ms_dispatch 0x7f8598000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.395+0000 7f858dffb640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.395+0000 7f858dffb640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.395+0000 7f858dffb640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T10:43:12.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.404+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85880504d0 con 0x7f859002cec0 2024-03-29T10:43:12.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.404+0000 7f858dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:12.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.404+0000 7f858dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:12.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.728+0000 7f857b7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:12.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.728+0000 7f857b7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:12.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.728+0000 7f857b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:12.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.728+0000 7f857b7fe640 1 -- 172.21.15.114:0/3504730724 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f859002e080 con 0x7f859002cec0 2024-03-29T10:43:12.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.729+0000 7f858dffb640 1 -- 172.21.15.114:0/3504730724 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f8588052400 con 0x7f859002cec0 2024-03-29T10:43:12.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.729+0000 7f858dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:12.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.729+0000 7f858dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:12.713 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:43:12.713 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:12.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.732+0000 7f857b7fe640 10 librados: watch_flush enter 2024-03-29T10:43:12.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.732+0000 7f857b7fe640 10 librados: watch_flush exit 2024-03-29T10:43:12.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.732+0000 7f857b7fe640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T10:43:12.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.732+0000 7f857b7fe640 10 client.4397.objecter successfully canceled tick 2024-03-29T10:43:12.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.732+0000 7f857b7fe640 1 -- 172.21.15.114:0/3504730724 >> v1:172.21.15.114:6816/775044307 conn(0x7f857c06de80 legacy=0x7f857c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:12.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.732+0000 7f857b7fe640 10 monclient: shutdown 2024-03-29T10:43:12.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.732+0000 7f857b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:12.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.732+0000 7f857b7fe640 1 -- 172.21.15.114:0/3504730724 >> v1:172.21.15.114:6790/0 conn(0x7f859002cec0 legacy=0x7f85900c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:12.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.733+0000 7f857b7fe640 1 -- 172.21.15.114:0/3504730724 shutdown_connections 2024-03-29T10:43:12.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.733+0000 7f857b7fe640 1 -- 172.21.15.114:0/3504730724 >> 172.21.15.114:0/3504730724 conn(0x7f859003a590 msgr2=0x7f85900c0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:12.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.735+0000 7f857b7fe640 1 -- 172.21.15.114:0/3504730724 shutdown_connections 2024-03-29T10:43:12.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.735+0000 7f857b7fe640 1 -- 172.21.15.114:0/3504730724 wait complete. 2024-03-29T10:43:12.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:12.735+0000 7f857b7fe640 1 librados: shutdown 2024-03-29T10:43:12.737 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:19.545 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:19.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.713+0000 7f3ff759e640 10 monclient: get_monmap_and_config 2024-03-29T10:43:19.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.713+0000 7f3ff759e640 10 monclient: build_initial_monmap 2024-03-29T10:43:19.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.713+0000 7f3ff759e640 10 monclient: monmap: 2024-03-29T10:43:19.697 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:19.697 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:19.697 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:43:19.714385+0000 2024-03-29T10:43:19.697 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:43:19.714385+0000 2024-03-29T10:43:19.697 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:19.698 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:19.698 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:19.698 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:43:19.698 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:43:19.698 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:19.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.717+0000 7f3ff759e640 1 Processor -- start 2024-03-29T10:43:19.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.717+0000 7f3ff759e640 1 -- start start 2024-03-29T10:43:19.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.717+0000 7f3ff759e640 10 monclient: init 2024-03-29T10:43:19.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.718+0000 7f3ff759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:19.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.718+0000 7f3ff759e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:19.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.718+0000 7f3ff759e640 10 monclient(hunting): picked mon.noname-b con 0x7f3ff0006a70 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:19.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.718+0000 7f3ff759e640 10 monclient(hunting): picked mon.noname-a con 0x7f3ff002b100 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:19.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.719+0000 7f3ff759e640 10 monclient(hunting): picked mon.noname-c con 0x7f3ff0029c30 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:19.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.719+0000 7f3ff759e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff002a270 con 0x7f3ff002b100 2024-03-29T10:43:19.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.719+0000 7f3ff759e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff002c440 con 0x7f3ff0029c30 2024-03-29T10:43:19.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.719+0000 7f3ff759e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff00c9440 con 0x7f3ff0006a70 2024-03-29T10:43:19.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.719+0000 7f3ff759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:19.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.719+0000 7f3ff759e640 10 monclient(hunting): authenticate will time out at 1064.960998s 2024-03-29T10:43:19.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.722+0000 7f3ff659c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f3ff0006a70 0x7f3ff00c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55083/0 (socket says 172.21.15.114:55083) 2024-03-29T10:43:19.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.722+0000 7f3ff659c640 1 -- 172.21.15.114:0/3955145753 learned_addr learned my addr 172.21.15.114:0/3955145753 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:19.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.724+0000 7f3ff559a640 1 -- 172.21.15.114:0/3955145753 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538858026 0 0) 0x7f3ff00c9440 con 0x7f3ff0006a70 2024-03-29T10:43:19.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.724+0000 7f3ff559a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:19.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.724+0000 7f3ff559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:19.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.724+0000 7f3ff559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:19.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.724+0000 7f3ff559a640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:43:19.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.724+0000 7f3ff559a640 1 -- 172.21.15.114:0/3955145753 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe0003520 con 0x7f3ff0006a70 2024-03-29T10:43:19.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 1 -- 172.21.15.114:0/3955145753 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723587765 0 0) 0x7f3ff002c440 con 0x7f3ff0029c30 2024-03-29T10:43:19.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:19.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:19.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:19.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:43:19.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 1 -- 172.21.15.114:0/3955145753 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe0003840 con 0x7f3ff0029c30 2024-03-29T10:43:19.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 1 -- 172.21.15.114:0/3955145753 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1995570747 0 0) 0x7f3fe0003520 con 0x7f3ff0006a70 2024-03-29T10:43:19.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 1 -- 172.21.15.114:0/3955145753 >> v1:172.21.15.176:6789/0 conn(0x7f3ff0029c30 legacy=0x7f3ff00d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:19.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 1 -- 172.21.15.114:0/3955145753 >> v1:172.21.15.114:6789/0 conn(0x7f3ff002b100 legacy=0x7f3ff00cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:19.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient: _finish_hunting 0 2024-03-29T10:43:19.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 1 monclient: found mon.noname-b 2024-03-29T10:43:19.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:19.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:43:19.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 1 -- 172.21.15.114:0/3955145753 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff00c95d0 con 0x7f3ff0006a70 2024-03-29T10:43:19.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient: _finish_auth 0 2024-03-29T10:43:19.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient: _check_auth_tickets 2024-03-29T10:43:19.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:19.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 1 -- 172.21.15.114:0/3955145753 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3ff804f730 con 0x7f3ff0006a70 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient: dump: 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:19.711 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff759e640 5 monclient: authenticate success, global_id 4400 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 1 -- 172.21.15.114:0/3955145753 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ff8075f10 con 0x7f3ff0006a70 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 1 -- 172.21.15.114:0/3955145753 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3ff80758b0 con 0x7f3ff0006a70 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.725+0000 7f3ff559a640 10 monclient: dump: 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:19.712 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:19.713 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:19.713 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:19.713 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:19.713 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:19.713 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.726+0000 7f3ff759e640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.726+0000 7f3ff759e640 4 set_mon_vals no callback set 2024-03-29T10:43:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.726+0000 7f3ff759e640 10 monclient: shutdown 2024-03-29T10:43:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.726+0000 7f3ff759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.726+0000 7f3ff759e640 1 -- 172.21.15.114:0/3955145753 >> v1:172.21.15.114:6790/0 conn(0x7f3ff0006a70 legacy=0x7f3ff00c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.727+0000 7f3ff759e640 1 -- 172.21.15.114:0/3955145753 shutdown_connections 2024-03-29T10:43:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.727+0000 7f3ff759e640 1 -- 172.21.15.114:0/3955145753 >> 172.21.15.114:0/3955145753 conn(0x7f3ff00058f0 msgr2=0x7f3ff0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:19.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.727+0000 7f3ff759e640 1 -- 172.21.15.114:0/3955145753 shutdown_connections 2024-03-29T10:43:19.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.727+0000 7f3ff759e640 1 -- 172.21.15.114:0/3955145753 wait complete. 2024-03-29T10:43:19.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.728+0000 7f3ff759e640 10 monclient: build_initial_monmap 2024-03-29T10:43:19.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.728+0000 7f3ff759e640 10 monclient: monmap: 2024-03-29T10:43:19.716 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:19.717 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:19.717 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:43:19.717 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:43:19.717 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:19.717 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:19.717 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:19.717 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:43:19.717 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:43:19.717 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:19.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.729+0000 7f3ff759e640 1 librados: starting msgr at 2024-03-29T10:43:19.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.729+0000 7f3ff759e640 1 librados: starting objecter 2024-03-29T10:43:19.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.729+0000 7f3ff759e640 1 Processor -- start 2024-03-29T10:43:19.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.732+0000 7f3ff759e640 1 -- start start 2024-03-29T10:43:19.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.732+0000 7f3ff759e640 1 librados: setting wanted keys 2024-03-29T10:43:19.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.732+0000 7f3ff759e640 1 librados: calling monclient init 2024-03-29T10:43:19.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.732+0000 7f3ff759e640 10 monclient: init 2024-03-29T10:43:19.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.733+0000 7f3ff759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:19.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.733+0000 7f3ff759e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:19.719 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.734+0000 7f3ff759e640 10 monclient(hunting): picked mon.noname-b con 0x7f3ff0006a70 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:19.719 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.736+0000 7f3ff659c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f3ff0006a70 0x7f3ff002d940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:50047/0 (socket says 172.21.15.114:50047) 2024-03-29T10:43:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.737+0000 7f3ff659c640 1 -- 172.21.15.114:0/3676972536 learned_addr learned my addr 172.21.15.114:0/3676972536 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.738+0000 7f3ff759e640 10 monclient(hunting): picked mon.noname-c con 0x7f3ff0029c30 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.738+0000 7f3ff759e640 10 monclient(hunting): picked mon.noname-a con 0x7f3ff0166ce0 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.738+0000 7f3fe6ffd640 10 client.?.objecter ms_handle_connect 0x7f3ff0006a70 2024-03-29T10:43:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.738+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff00caad0 con 0x7f3ff0166ce0 2024-03-29T10:43:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.738+0000 7f3fe6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.738+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff002f090 con 0x7f3ff0006a70 2024-03-29T10:43:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.738+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff002f300 con 0x7f3ff0029c30 2024-03-29T10:43:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.738+0000 7f3ff759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.738+0000 7f3ff759e640 10 monclient(hunting): authenticate will time out at 1064.979710s 2024-03-29T10:43:19.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.739+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100665152 0 0) 0x7f3ff002f090 con 0x7f3ff0006a70 2024-03-29T10:43:19.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.739+0000 7f3fe6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:19.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.739+0000 7f3fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:19.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.740+0000 7f3fe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:19.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.740+0000 7f3fe6ffd640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:43:19.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.740+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd8004eb0 con 0x7f3ff0006a70 2024-03-29T10:43:19.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.740+0000 7f3fe6ffd640 10 client.?.objecter ms_handle_connect 0x7f3ff0029c30 2024-03-29T10:43:19.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.740+0000 7f3fe6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:19.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.740+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 362557732 0 0) 0x7f3fd8004eb0 con 0x7f3ff0006a70 2024-03-29T10:43:19.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.740+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ff002f090 con 0x7f3ff0006a70 2024-03-29T10:43:19.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.740+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267471661 0 0) 0x7f3ff002f300 con 0x7f3ff0029c30 2024-03-29T10:43:19.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.741+0000 7f3fe6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.741+0000 7f3fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.741+0000 7f3fe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.741+0000 7f3fe6ffd640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.741+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd8004eb0 con 0x7f3ff0029c30 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 50372128 0 0) 0x7f3fd8004eb0 con 0x7f3ff0029c30 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ff002f300 con 0x7f3ff0029c30 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3fe8001830 con 0x7f3ff0029c30 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 10 monclient(hunting): dump: 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:19.728 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3311108167 0 0) 0x7f3ff002f300 con 0x7f3ff0029c30 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 >> v1:172.21.15.176:6789/0 conn(0x7f3ff0006a70 legacy=0x7f3ff002d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 >> v1:172.21.15.114:6789/0 conn(0x7f3ff0166ce0 legacy=0x7f3ff016ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 1 monclient: found mon.c 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff002e050 con 0x7f3ff0029c30 2024-03-29T10:43:19.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:19.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:19.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.742+0000 7f3fe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:19.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.743+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fe8001d40 con 0x7f3ff0029c30 2024-03-29T10:43:19.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.743+0000 7f3fe6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:19.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.743+0000 7f3ff4d99640 4 set_mon_vals no callback set 2024-03-29T10:43:19.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.744+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f3fe80037b0 con 0x7f3ff0029c30 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.744+0000 7f3fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.744+0000 7f3fe6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.744+0000 7f3fe6ffd640 10 monclient: dump: 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.745+0000 7f3ff759e640 5 monclient: authenticate success, global_id 4403 2024-03-29T10:43:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.745+0000 7f3ff759e640 10 monclient: _renew_subs 2024-03-29T10:43:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.745+0000 7f3ff759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.745+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff002cc30 con 0x7f3ff0029c30 2024-03-29T10:43:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.745+0000 7f3ff759e640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.745+0000 7f3ff759e640 10 monclient: _renew_subs 2024-03-29T10:43:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.745+0000 7f3ff759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.745+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff016d700 con 0x7f3ff0029c30 2024-03-29T10:43:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.745+0000 7f3ff759e640 1 librados: init done 2024-03-29T10:43:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.747+0000 7f3ff759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.747+0000 7f3ff759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.747+0000 7f3ff759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.747+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff01670f0 con 0x7f3ff0029c30 2024-03-29T10:43:19.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.751+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f3fe8019780 con 0x7f3ff0029c30 2024-03-29T10:43:19.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.751+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f3fe8083750 con 0x7f3ff0029c30 2024-03-29T10:43:19.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.751+0000 7f3fe6ffd640 10 client.4403.objecter ms_dispatch 0x7f3ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:19.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.751+0000 7f3ff659c640 10 monclient: get_auth_request con 0x7f3fd806dd10 auth_method 0 2024-03-29T10:43:19.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.751+0000 7f3fe6ffd640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:19.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.751+0000 7f3fe6ffd640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:19.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.752+0000 7f3fe6ffd640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T10:43:19.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.752+0000 7f3fe6ffd640 10 client.4403.objecter ms_handle_connect 0x7f3fd806dd10 2024-03-29T10:43:19.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.761+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3fe804ee60 con 0x7f3ff0029c30 2024-03-29T10:43:19.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.761+0000 7f3fe6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:19.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:19.761+0000 7f3fe6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:20.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.078+0000 7f3ff759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:20.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.078+0000 7f3ff759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:20.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.078+0000 7f3ff759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:20.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.078+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ff0004cd0 con 0x7f3ff0029c30 2024-03-29T10:43:20.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.080+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/3676972536 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f3fe8050f60 con 0x7f3ff0029c30 2024-03-29T10:43:20.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.080+0000 7f3fe6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:20.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.080+0000 7f3fe6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:20.063 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:43:20.063 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:20.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.083+0000 7f3ff759e640 10 librados: watch_flush enter 2024-03-29T10:43:20.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.085+0000 7f3ff759e640 10 librados: watch_flush exit 2024-03-29T10:43:20.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.085+0000 7f3ff759e640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T10:43:20.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.085+0000 7f3ff759e640 10 client.4403.objecter successfully canceled tick 2024-03-29T10:43:20.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.090+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 >> v1:172.21.15.114:6816/775044307 conn(0x7f3fd806dd10 legacy=0x7f3fd80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:20.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.090+0000 7f3ff759e640 10 monclient: shutdown 2024-03-29T10:43:20.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.090+0000 7f3ff759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:20.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.090+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 >> v1:172.21.15.114:6790/0 conn(0x7f3ff0029c30 legacy=0x7f3ff01665d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:20.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.091+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 shutdown_connections 2024-03-29T10:43:20.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.091+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 >> 172.21.15.114:0/3676972536 conn(0x7f3ff00058f0 msgr2=0x7f3ff00c8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:20.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.094+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 shutdown_connections 2024-03-29T10:43:20.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.094+0000 7f3ff759e640 1 -- 172.21.15.114:0/3676972536 wait complete. 2024-03-29T10:43:20.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:20.094+0000 7f3ff759e640 1 librados: shutdown 2024-03-29T10:43:20.095 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:26.902 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:27.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.080+0000 7fd980a4a640 10 monclient: get_monmap_and_config 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.080+0000 7fd980a4a640 10 monclient: build_initial_monmap 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.080+0000 7fd980a4a640 10 monclient: monmap: 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:43:27.081609+0000 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:43:27.081609+0000 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:43:27.064 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:27.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.081+0000 7fd980a4a640 1 Processor -- start 2024-03-29T10:43:27.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.081+0000 7fd980a4a640 1 -- start start 2024-03-29T10:43:27.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.081+0000 7fd980a4a640 10 monclient: init 2024-03-29T10:43:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.082+0000 7fd980a4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.082+0000 7fd980a4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.082+0000 7fd980a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fd97402b4c0 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.082+0000 7fd980a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fd97400b8a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.082+0000 7fd980a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fd97402a5a0 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.082+0000 7fd980a4a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97402c5c0 con 0x7fd97400b8a0 2024-03-29T10:43:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.082+0000 7fd980a4a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740c11a0 con 0x7fd97402b4c0 2024-03-29T10:43:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.082+0000 7fd980a4a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97402afa0 con 0x7fd97402a5a0 2024-03-29T10:43:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.082+0000 7fd980a4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.082+0000 7fd980a4a640 10 monclient(hunting): authenticate will time out at 1072.323528s 2024-03-29T10:43:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.085+0000 7fd979d9b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd97400b8a0 0x7fd9740c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:53534/0 (socket says 172.21.15.114:53534) 2024-03-29T10:43:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.085+0000 7fd979d9b640 1 -- 172.21.15.114:0/2804635056 learned_addr learned my addr 172.21.15.114:0/2804635056 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879253608 0 0) 0x7fd97402afa0 con 0x7fd97402a5a0 2024-03-29T10:43:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:43:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd964003520 con 0x7fd97402a5a0 2024-03-29T10:43:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484909992 0 0) 0x7fd9740c11a0 con 0x7fd97402b4c0 2024-03-29T10:43:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:27.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:27.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:43:27.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd97402afa0 con 0x7fd97402b4c0 2024-03-29T10:43:27.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995348027 0 0) 0x7fd97402c5c0 con 0x7fd97400b8a0 2024-03-29T10:43:27.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:27.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:27.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:27.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:43:27.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.086+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9740c11a0 con 0x7fd97400b8a0 2024-03-29T10:43:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3151858897 0 0) 0x7fd97402afa0 con 0x7fd97402b4c0 2024-03-29T10:43:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 >> v1:172.21.15.114:6790/0 conn(0x7fd97402a5a0 legacy=0x7fd9740c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 >> v1:172.21.15.114:6789/0 conn(0x7fd97400b8a0 legacy=0x7fd9740c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 10 monclient: _finish_hunting 0 2024-03-29T10:43:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 1 monclient: found mon.noname-c 2024-03-29T10:43:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9740cb160 con 0x7fd97402b4c0 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 10 monclient: _finish_auth 0 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 10 monclient: _check_auth_tickets 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd970001670 con 0x7fd97402b4c0 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 10 monclient: dump: 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:27.074 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd980a4a640 5 monclient: authenticate success, global_id 4360 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd980a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9700017d0 con 0x7fd97402b4c0 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 1 -- 172.21.15.114:0/2804635056 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd970001b60 con 0x7fd97402b4c0 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd97959a640 10 monclient: dump: 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:27.075 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd980a4a640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.087+0000 7fd980a4a640 4 set_mon_vals no callback set 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.088+0000 7fd980a4a640 10 monclient: shutdown 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.088+0000 7fd980a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:27.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.088+0000 7fd980a4a640 1 -- 172.21.15.114:0/2804635056 >> v1:172.21.15.176:6789/0 conn(0x7fd97402b4c0 legacy=0x7fd974006890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:27.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.090+0000 7fd980a4a640 1 -- 172.21.15.114:0/2804635056 shutdown_connections 2024-03-29T10:43:27.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.090+0000 7fd980a4a640 1 -- 172.21.15.114:0/2804635056 >> 172.21.15.114:0/2804635056 conn(0x7fd9740058f0 msgr2=0x7fd974005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:27.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.091+0000 7fd980a4a640 1 -- 172.21.15.114:0/2804635056 shutdown_connections 2024-03-29T10:43:27.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.091+0000 7fd980a4a640 1 -- 172.21.15.114:0/2804635056 wait complete. 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.092+0000 7fd980a4a640 10 monclient: build_initial_monmap 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.092+0000 7fd980a4a640 10 monclient: monmap: 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:27.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.092+0000 7fd980a4a640 1 librados: starting msgr at 2024-03-29T10:43:27.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.092+0000 7fd980a4a640 1 librados: starting objecter 2024-03-29T10:43:27.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.093+0000 7fd980a4a640 1 Processor -- start 2024-03-29T10:43:27.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.093+0000 7fd980a4a640 1 -- start start 2024-03-29T10:43:27.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.093+0000 7fd980a4a640 1 librados: setting wanted keys 2024-03-29T10:43:27.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.093+0000 7fd980a4a640 1 librados: calling monclient init 2024-03-29T10:43:27.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.093+0000 7fd980a4a640 10 monclient: init 2024-03-29T10:43:27.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.094+0000 7fd980a4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:27.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.094+0000 7fd980a4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:27.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.094+0000 7fd980a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fd97400b8a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:27.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.097+0000 7fd980a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fd97402a5a0 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:27.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.097+0000 7fd980a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fd97402b4c0 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:27.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.097+0000 7fd980a4a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740b5960 con 0x7fd97400b8a0 2024-03-29T10:43:27.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.097+0000 7fd980a4a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740b5c40 con 0x7fd97402b4c0 2024-03-29T10:43:27.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.097+0000 7fd980a4a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740b6ad0 con 0x7fd97402a5a0 2024-03-29T10:43:27.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.097+0000 7fd980a4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:27.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.097+0000 7fd980a4a640 10 monclient(hunting): authenticate will time out at 1072.338450s 2024-03-29T10:43:27.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.097+0000 7fd979d9b640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fd97402a5a0 0x7fd9740c0850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:45175/0 (socket says 172.21.15.114:45175) 2024-03-29T10:43:27.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.097+0000 7fd979d9b640 1 -- 172.21.15.114:0/1595442819 learned_addr learned my addr 172.21.15.114:0/1595442819 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:27.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.097+0000 7fd96affd640 10 client.?.objecter ms_handle_connect 0x7fd97402a5a0 2024-03-29T10:43:27.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.098+0000 7fd96affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:27.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.098+0000 7fd96affd640 10 client.?.objecter ms_handle_connect 0x7fd97400b8a0 2024-03-29T10:43:27.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.098+0000 7fd96affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:27.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.098+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138443489 0 0) 0x7fd9740b6ad0 con 0x7fd97402a5a0 2024-03-29T10:43:27.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.098+0000 7fd96affd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:27.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.098+0000 7fd96affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:27.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.098+0000 7fd96affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:27.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.098+0000 7fd96affd640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:43:27.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.098+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd95c004eb0 con 0x7fd97402a5a0 2024-03-29T10:43:27.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.098+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322635420 0 0) 0x7fd9740b5960 con 0x7fd97400b8a0 2024-03-29T10:43:27.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.098+0000 7fd96affd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:27.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.099+0000 7fd96affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:27.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.099+0000 7fd96affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:27.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.099+0000 7fd96affd640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:43:27.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.099+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9740b6ad0 con 0x7fd97400b8a0 2024-03-29T10:43:27.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.100+0000 7fd96affd640 10 client.?.objecter ms_handle_connect 0x7fd97402b4c0 2024-03-29T10:43:27.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.100+0000 7fd96affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:27.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.101+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338086427 0 0) 0x7fd9740b5c40 con 0x7fd97402b4c0 2024-03-29T10:43:27.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.101+0000 7fd96affd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:27.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.101+0000 7fd96affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:27.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.101+0000 7fd96affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:27.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.101+0000 7fd96affd640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:43:27.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.101+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9740b5960 con 0x7fd97402b4c0 2024-03-29T10:43:27.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.101+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 52899394 0 0) 0x7fd9740b6ad0 con 0x7fd97400b8a0 2024-03-29T10:43:27.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.101+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9740b5c40 con 0x7fd97400b8a0 2024-03-29T10:43:27.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.101+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd970001ae0 con 0x7fd97400b8a0 2024-03-29T10:43:27.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.101+0000 7fd96affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:27.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:27.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 10 monclient(hunting): dump: 2024-03-29T10:43:27.090 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:27.090 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:27.090 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:27.090 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:27.090 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3461005002 0 0) 0x7fd9740b5960 con 0x7fd97402b4c0 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9740b6ad0 con 0x7fd97402b4c0 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd96c001880 con 0x7fd97402b4c0 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 10 monclient(hunting): dump: 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:27.091 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3773838638 0 0) 0x7fd9740b6ad0 con 0x7fd97402b4c0 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 >> v1:172.21.15.114:6790/0 conn(0x7fd97402a5a0 legacy=0x7fd9740c0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 >> v1:172.21.15.114:6789/0 conn(0x7fd97400b8a0 legacy=0x7fd9740c15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 1 monclient: found mon.b 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:27.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.102+0000 7fd96affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9740b6160 con 0x7fd97402b4c0 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd96affd640 10 monclient: _finish_auth 0 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd96affd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd96affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd980a4a640 5 monclient: authenticate success, global_id 4363 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd980a4a640 10 monclient: _renew_subs 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd980a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd980a4a640 1 -- 172.21.15.114:0/1595442819 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9740b5580 con 0x7fd97402b4c0 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd980a4a640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd980a4a640 10 monclient: _renew_subs 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd980a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd980a4a640 1 -- 172.21.15.114:0/1595442819 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9740cae10 con 0x7fd97402b4c0 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.103+0000 7fd980a4a640 1 librados: init done 2024-03-29T10:43:27.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.104+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd96c001d50 con 0x7fd97402b4c0 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.104+0000 7fd96affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.104+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd96c0037b0 con 0x7fd97402b4c0 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.104+0000 7fd96affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.104+0000 7fd96affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.104+0000 7fd96affd640 10 monclient: dump: 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:27.094 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:27.095 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:27.095 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:27.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.104+0000 7fd968ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:27.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.104+0000 7fd968ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:27.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.104+0000 7fd968ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:27.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.104+0000 7fd968ff9640 1 -- 172.21.15.114:0/1595442819 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd974033040 con 0x7fd97402b4c0 2024-03-29T10:43:27.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.106+0000 7fd96bfff640 4 set_mon_vals no callback set 2024-03-29T10:43:27.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.109+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fd96c019780 con 0x7fd97402b4c0 2024-03-29T10:43:27.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.109+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fd96c083610 con 0x7fd97402b4c0 2024-03-29T10:43:27.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.109+0000 7fd96affd640 10 client.4363.objecter ms_dispatch 0x7fd97c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:27.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.110+0000 7fd96affd640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:27.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.110+0000 7fd96affd640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:27.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.110+0000 7fd96affd640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-29T10:43:27.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.110+0000 7fd97a59c640 10 monclient: get_auth_request con 0x7fd95c06df00 auth_method 0 2024-03-29T10:43:27.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.110+0000 7fd96affd640 10 client.4363.objecter ms_handle_connect 0x7fd95c06df00 2024-03-29T10:43:27.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.113+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd96c04ed20 con 0x7fd97402b4c0 2024-03-29T10:43:27.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.113+0000 7fd96affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:27.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.113+0000 7fd96affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:27.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.467+0000 7fd968ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:27.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.467+0000 7fd968ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:27.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.467+0000 7fd968ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:27.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.467+0000 7fd968ff9640 1 -- 172.21.15.114:0/1595442819 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd974032a50 con 0x7fd97402b4c0 2024-03-29T10:43:27.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.470+0000 7fd96affd640 1 -- 172.21.15.114:0/1595442819 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fd96c050e20 con 0x7fd97402b4c0 2024-03-29T10:43:27.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.470+0000 7fd96affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:27.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.470+0000 7fd96affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:27.454 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:43:27.454 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:27.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.474+0000 7fd968ff9640 10 librados: watch_flush enter 2024-03-29T10:43:27.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.474+0000 7fd968ff9640 10 librados: watch_flush exit 2024-03-29T10:43:27.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.474+0000 7fd968ff9640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-29T10:43:27.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.474+0000 7fd968ff9640 10 client.4363.objecter successfully canceled tick 2024-03-29T10:43:27.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.474+0000 7fd968ff9640 1 -- 172.21.15.114:0/1595442819 >> v1:172.21.15.114:6816/775044307 conn(0x7fd95c06df00 legacy=0x7fd95c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:27.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.475+0000 7fd968ff9640 10 monclient: shutdown 2024-03-29T10:43:27.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.475+0000 7fd968ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:27.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.475+0000 7fd968ff9640 1 -- 172.21.15.114:0/1595442819 >> v1:172.21.15.176:6789/0 conn(0x7fd97402b4c0 legacy=0x7fd97416c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:27.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.476+0000 7fd968ff9640 1 -- 172.21.15.114:0/1595442819 shutdown_connections 2024-03-29T10:43:27.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.476+0000 7fd968ff9640 1 -- 172.21.15.114:0/1595442819 >> 172.21.15.114:0/1595442819 conn(0x7fd9740058f0 msgr2=0x7fd974034c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:27.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.478+0000 7fd968ff9640 1 -- 172.21.15.114:0/1595442819 shutdown_connections 2024-03-29T10:43:27.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.478+0000 7fd968ff9640 1 -- 172.21.15.114:0/1595442819 wait complete. 2024-03-29T10:43:27.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:27.478+0000 7fd968ff9640 1 librados: shutdown 2024-03-29T10:43:27.480 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:34.287 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:34.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.462+0000 7ff3c7a63640 10 monclient: get_monmap_and_config 2024-03-29T10:43:34.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.462+0000 7ff3c7a63640 10 monclient: build_initial_monmap 2024-03-29T10:43:34.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.462+0000 7ff3c7a63640 10 monclient: monmap: 2024-03-29T10:43:34.445 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:34.445 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:34.445 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:43:34.462740+0000 2024-03-29T10:43:34.445 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:43:34.462740+0000 2024-03-29T10:43:34.445 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:34.445 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:34.446 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:34.446 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:43:34.446 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:43:34.446 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:34.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.463+0000 7ff3c7a63640 1 Processor -- start 2024-03-29T10:43:34.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.464+0000 7ff3c7a63640 1 -- start start 2024-03-29T10:43:34.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.464+0000 7ff3c7a63640 10 monclient: init 2024-03-29T10:43:34.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.465+0000 7ff3c7a63640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:34.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.465+0000 7ff3c7a63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.465+0000 7ff3c7a63640 10 monclient(hunting): picked mon.noname-b con 0x7ff3c01422b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.465+0000 7ff3c7a63640 10 monclient(hunting): picked mon.noname-c con 0x7ff3c0142690 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.465+0000 7ff3c7a63640 10 monclient(hunting): picked mon.noname-a con 0x7ff3c00b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.465+0000 7ff3c57ff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff3c01422b0 0x7ff3c013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:41698/0 (socket says 172.21.15.114:41698) 2024-03-29T10:43:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.466+0000 7ff3c57ff640 1 -- 172.21.15.114:0/4285330360 learned_addr learned my addr 172.21.15.114:0/4285330360 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.466+0000 7ff3c7a63640 1 -- 172.21.15.114:0/4285330360 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c0142c80 con 0x7ff3c00b17d0 2024-03-29T10:43:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.466+0000 7ff3c7a63640 1 -- 172.21.15.114:0/4285330360 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00aae40 con 0x7ff3c0142690 2024-03-29T10:43:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.466+0000 7ff3c7a63640 1 -- 172.21.15.114:0/4285330360 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c013fbd0 con 0x7ff3c01422b0 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.466+0000 7ff3c7a63640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.466+0000 7ff3c7a63640 10 monclient(hunting): authenticate will time out at 1079.706575s 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.466+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537989091 0 0) 0x7ff3c013fbd0 con 0x7ff3c01422b0 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ac003520 con 0x7ff3c01422b0 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082427501 0 0) 0x7ff3c00aae40 con 0x7ff3c0142690 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:43:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c013fbd0 con 0x7ff3c0142690 2024-03-29T10:43:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260558841 0 0) 0x7ff3c0142c80 con 0x7ff3c00b17d0 2024-03-29T10:43:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.467+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c00aae40 con 0x7ff3c00b17d0 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 706793835 0 0) 0x7ff3c00aae40 con 0x7ff3c00b17d0 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 >> v1:172.21.15.114:6790/0 conn(0x7ff3c01422b0 legacy=0x7ff3c013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 >> v1:172.21.15.176:6789/0 conn(0x7ff3c0142690 legacy=0x7ff3c00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 1 monclient: found mon.noname-a 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c014f920 con 0x7ff3c00b17d0 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 10 monclient: _finish_auth 0 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff3b80017e0 con 0x7ff3c00b17d0 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 10 monclient: dump: 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:34.456 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3c7a63640 5 monclient: authenticate success, global_id 4395 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3c7a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3b8001940 con 0x7ff3c00b17d0 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 1 -- 172.21.15.114:0/4285330360 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff3b8001cd0 con 0x7ff3c00b17d0 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3b7fff640 10 monclient: dump: 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:34.457 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3c7a63640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.468+0000 7ff3c7a63640 4 set_mon_vals no callback set 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.469+0000 7ff3c7a63640 10 monclient: shutdown 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.469+0000 7ff3c7a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.469+0000 7ff3c7a63640 1 -- 172.21.15.114:0/4285330360 >> v1:172.21.15.114:6789/0 conn(0x7ff3c00b17d0 legacy=0x7ff3c00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.470+0000 7ff3c7a63640 1 -- 172.21.15.114:0/4285330360 shutdown_connections 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.470+0000 7ff3c7a63640 1 -- 172.21.15.114:0/4285330360 >> 172.21.15.114:0/4285330360 conn(0x7ff3c0066a00 msgr2=0x7ff3c0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:34.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.470+0000 7ff3c7a63640 1 -- 172.21.15.114:0/4285330360 shutdown_connections 2024-03-29T10:43:34.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.470+0000 7ff3c7a63640 1 -- 172.21.15.114:0/4285330360 wait complete. 2024-03-29T10:43:34.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.471+0000 7ff3c7a63640 10 monclient: build_initial_monmap 2024-03-29T10:43:34.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.471+0000 7ff3c7a63640 10 monclient: monmap: 2024-03-29T10:43:34.459 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:34.459 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:34.459 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:43:34.459 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.471+0000 7ff3c7a63640 1 librados: starting msgr at 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.471+0000 7ff3c7a63640 1 librados: starting objecter 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.472+0000 7ff3c7a63640 1 Processor -- start 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.472+0000 7ff3c7a63640 1 -- start start 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.472+0000 7ff3c7a63640 1 librados: setting wanted keys 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.472+0000 7ff3c7a63640 1 librados: calling monclient init 2024-03-29T10:43:34.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.472+0000 7ff3c7a63640 10 monclient: init 2024-03-29T10:43:34.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.473+0000 7ff3c7a63640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:34.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.473+0000 7ff3c7a63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:34.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.473+0000 7ff3c7a63640 10 monclient(hunting): picked mon.noname-a con 0x7ff3c00b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:34.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.473+0000 7ff3c57ff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff3c00b17d0 0x7ff3c00b0970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48554/0 (socket says 172.21.15.114:48554) 2024-03-29T10:43:34.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.473+0000 7ff3c7a63640 10 monclient(hunting): picked mon.noname-c con 0x7ff3c01422b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:34.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.473+0000 7ff3c7a63640 10 monclient(hunting): picked mon.noname-b con 0x7ff3c0142690 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.473+0000 7ff3c7a63640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00acf30 con 0x7ff3c00b17d0 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.473+0000 7ff3c7a63640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00ad160 con 0x7ff3c0142690 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.473+0000 7ff3c7a63640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00ac050 con 0x7ff3c01422b0 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.473+0000 7ff3c7a63640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.473+0000 7ff3c7a63640 10 monclient(hunting): authenticate will time out at 1079.713849s 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3c57ff640 1 -- 172.21.15.114:0/242970099 learned_addr learned my addr 172.21.15.114:0/242970099 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 client.?.objecter ms_handle_connect 0x7ff3c0142690 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 client.?.objecter ms_handle_connect 0x7ff3c00b17d0 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 client.?.objecter ms_handle_connect 0x7ff3c01422b0 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407363279 0 0) 0x7ff3c00acf30 con 0x7ff3c00b17d0 2024-03-29T10:43:34.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:34.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:34.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:34.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:43:34.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a4004eb0 con 0x7ff3c00b17d0 2024-03-29T10:43:34.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080854382 0 0) 0x7ff3c00ac050 con 0x7ff3c01422b0 2024-03-29T10:43:34.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:34.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:34.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:34.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:43:34.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c00acf30 con 0x7ff3c01422b0 2024-03-29T10:43:34.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.474+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3737852101 0 0) 0x7ff3a4004eb0 con 0x7ff3c00b17d0 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3c00ac050 con 0x7ff3c00b17d0 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff3bc001950 con 0x7ff3c00b17d0 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient(hunting): dump: 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:34.467 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3474706344 0 0) 0x7ff3c00ac050 con 0x7ff3c00b17d0 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 >> v1:172.21.15.114:6790/0 conn(0x7ff3c01422b0 legacy=0x7ff3c012e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 >> v1:172.21.15.176:6789/0 conn(0x7ff3c0142690 legacy=0x7ff3c01e73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 1 monclient: found mon.a 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c0071e10 con 0x7ff3c00b17d0 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3c7a63640 5 monclient: authenticate success, global_id 4398 2024-03-29T10:43:34.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3c7a63640 10 monclient: _renew_subs 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3c7a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3c7a63640 1 -- 172.21.15.114:0/242970099 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c0071880 con 0x7ff3c00b17d0 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3bc001ab0 con 0x7ff3c00b17d0 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff3bc004a90 con 0x7ff3c00b17d0 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b5ffb640 10 monclient: dump: 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:34.469 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3b77fe640 4 set_mon_vals no callback set 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3c7a63640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3c7a63640 10 monclient: _renew_subs 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3c7a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3c7a63640 1 -- 172.21.15.114:0/242970099 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c00b1080 con 0x7ff3c00b17d0 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.475+0000 7ff3c7a63640 1 librados: init done 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.477+0000 7ff3c7a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.477+0000 7ff3c7a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:34.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.477+0000 7ff3c7a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:43:34.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.477+0000 7ff3c7a63640 1 -- 172.21.15.114:0/242970099 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c012e4f0 con 0x7ff3c00b17d0 2024-03-29T10:43:34.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.478+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7ff3bc004bf0 con 0x7ff3c00b17d0 2024-03-29T10:43:34.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.479+0000 7ff3c4ffe640 10 monclient: get_auth_request con 0x7ff3a406df50 auth_method 0 2024-03-29T10:43:34.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.479+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7ff3bc084b80 con 0x7ff3c00b17d0 2024-03-29T10:43:34.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.479+0000 7ff3b5ffb640 10 client.4398.objecter ms_dispatch 0x7ff3c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:34.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.479+0000 7ff3b5ffb640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:34.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.479+0000 7ff3b5ffb640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:34.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.479+0000 7ff3b5ffb640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-29T10:43:34.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.479+0000 7ff3b5ffb640 10 client.4398.objecter ms_handle_connect 0x7ff3a406df50 2024-03-29T10:43:34.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.488+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3bc050290 con 0x7ff3c00b17d0 2024-03-29T10:43:34.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.488+0000 7ff3b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:34.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.488+0000 7ff3b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:34.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.858+0000 7ff3c7a63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:34.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.858+0000 7ff3c7a63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:34.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.858+0000 7ff3c7a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:43:34.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.858+0000 7ff3c7a63640 1 -- 172.21.15.114:0/242970099 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3c0075f10 con 0x7ff3c00b17d0 2024-03-29T10:43:34.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.860+0000 7ff3b5ffb640 1 -- 172.21.15.114:0/242970099 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7ff3bc052390 con 0x7ff3c00b17d0 2024-03-29T10:43:34.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.860+0000 7ff3b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:34.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.860+0000 7ff3b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:34.842 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:43:34.842 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:34.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.863+0000 7ff3c7a63640 10 librados: watch_flush enter 2024-03-29T10:43:34.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.863+0000 7ff3c7a63640 10 librados: watch_flush exit 2024-03-29T10:43:34.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.863+0000 7ff3c7a63640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-29T10:43:34.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.863+0000 7ff3c7a63640 10 client.4398.objecter successfully canceled tick 2024-03-29T10:43:34.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.864+0000 7ff3c7a63640 1 -- 172.21.15.114:0/242970099 >> v1:172.21.15.114:6816/775044307 conn(0x7ff3a406df50 legacy=0x7ff3a4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.864+0000 7ff3c7a63640 10 monclient: shutdown 2024-03-29T10:43:34.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.864+0000 7ff3c7a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:34.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.864+0000 7ff3c7a63640 1 -- 172.21.15.114:0/242970099 >> v1:172.21.15.114:6789/0 conn(0x7ff3c00b17d0 legacy=0x7ff3c00b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.864+0000 7ff3c7a63640 1 -- 172.21.15.114:0/242970099 shutdown_connections 2024-03-29T10:43:34.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.864+0000 7ff3c7a63640 1 -- 172.21.15.114:0/242970099 >> 172.21.15.114:0/242970099 conn(0x7ff3c0066a00 msgr2=0x7ff3c013f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:34.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.864+0000 7ff3c7a63640 1 -- 172.21.15.114:0/242970099 shutdown_connections 2024-03-29T10:43:34.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.864+0000 7ff3c7a63640 1 -- 172.21.15.114:0/242970099 wait complete. 2024-03-29T10:43:34.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:34.864+0000 7ff3c7a63640 1 librados: shutdown 2024-03-29T10:43:34.862 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:41.668 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:41.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.842+0000 7fa7a7fac640 10 monclient: get_monmap_and_config 2024-03-29T10:43:41.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.842+0000 7fa7a7fac640 10 monclient: build_initial_monmap 2024-03-29T10:43:41.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.843+0000 7fa7a7fac640 10 monclient: monmap: 2024-03-29T10:43:41.825 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:41.825 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:41.825 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:43:41.843269+0000 2024-03-29T10:43:41.825 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:43:41.843269+0000 2024-03-29T10:43:41.826 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:41.826 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:41.826 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:41.826 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:43:41.826 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:43:41.826 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:41.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.847+0000 7fa7a7fac640 1 Processor -- start 2024-03-29T10:43:41.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.848+0000 7fa7a7fac640 1 -- start start 2024-03-29T10:43:41.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.848+0000 7fa7a7fac640 10 monclient: init 2024-03-29T10:43:41.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.849+0000 7fa7a7fac640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.849+0000 7fa7a7fac640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.849+0000 7fa7a7fac640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a006d9d0 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.849+0000 7fa7a7fac640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a013ef30 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.849+0000 7fa7a7fac640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a00b07d0 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.849+0000 7fa7a7fac640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a00aed10 con 0x7fa7a00b07d0 2024-03-29T10:43:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.849+0000 7fa7a7fac640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a00b0c40 con 0x7fa7a006d9d0 2024-03-29T10:43:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.849+0000 7fa7a7fac640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a00ab1b0 con 0x7fa7a013ef30 2024-03-29T10:43:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.849+0000 7fa7a7fac640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.849+0000 7fa7a7fac640 10 monclient(hunting): authenticate will time out at 1087.089160s 2024-03-29T10:43:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.853+0000 7fa7a5547640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fa7a013ef30 0x7fa7a00af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:47860/0 (socket says 172.21.15.114:47860) 2024-03-29T10:43:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.853+0000 7fa7a5547640 1 -- 172.21.15.114:0/3105165838 learned_addr learned my addr 172.21.15.114:0/3105165838 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:41.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 1 -- 172.21.15.114:0/3105165838 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575673702 0 0) 0x7fa7a00b0c40 con 0x7fa7a006d9d0 2024-03-29T10:43:41.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient(hunting): _negotiate 2024-03-29T10:43:41.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 1 -- 172.21.15.114:0/3105165838 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa78c003520 con 0x7fa7a006d9d0 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 1 -- 172.21.15.114:0/3105165838 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723473662 0 0) 0x7fa7a00ab1b0 con 0x7fa7a013ef30 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient(hunting): _negotiate 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 1 -- 172.21.15.114:0/3105165838 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a00b0c40 con 0x7fa7a013ef30 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 1 -- 172.21.15.114:0/3105165838 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 335407652 0 0) 0x7fa78c003520 con 0x7fa7a006d9d0 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 1 -- 172.21.15.114:0/3105165838 >> v1:172.21.15.114:6790/0 conn(0x7fa7a013ef30 legacy=0x7fa7a00af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 1 -- 172.21.15.114:0/3105165838 >> v1:172.21.15.114:6789/0 conn(0x7fa7a00b07d0 legacy=0x7fa7a014bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient: _finish_hunting 0 2024-03-29T10:43:41.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 1 monclient: found mon.noname-c 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 1 -- 172.21.15.114:0/3105165838 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a00ab490 con 0x7fa7a006d9d0 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient: _finish_auth 0 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 10 monclient: _check_auth_tickets 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.854+0000 7fa7a4d46640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.855+0000 7fa7a7fac640 5 monclient: authenticate success, global_id 4372 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.855+0000 7fa7a7fac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.855+0000 7fa7a4d46640 1 -- 172.21.15.114:0/3105165838 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa790001540 con 0x7fa7a006d9d0 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.855+0000 7fa7a4d46640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.855+0000 7fa7a4d46640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.855+0000 7fa7a4d46640 10 monclient: dump: 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:41.841 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.855+0000 7fa7a7fac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.855+0000 7fa7a4d46640 1 -- 172.21.15.114:0/3105165838 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa790001760 con 0x7fa7a006d9d0 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.855+0000 7fa7a4d46640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.855+0000 7fa7a7fac640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.855+0000 7fa7a7fac640 4 set_mon_vals no callback set 2024-03-29T10:43:41.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.856+0000 7fa7a7fac640 10 monclient: shutdown 2024-03-29T10:43:41.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.856+0000 7fa7a7fac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:41.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.856+0000 7fa7a7fac640 1 -- 172.21.15.114:0/3105165838 >> v1:172.21.15.176:6789/0 conn(0x7fa7a006d9d0 legacy=0x7fa7a013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:41.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.858+0000 7fa7a7fac640 1 -- 172.21.15.114:0/3105165838 shutdown_connections 2024-03-29T10:43:41.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.858+0000 7fa7a7fac640 1 -- 172.21.15.114:0/3105165838 >> 172.21.15.114:0/3105165838 conn(0x7fa7a006b660 msgr2=0x7fa7a006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:41.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.858+0000 7fa7a7fac640 1 -- 172.21.15.114:0/3105165838 shutdown_connections 2024-03-29T10:43:41.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.858+0000 7fa7a7fac640 1 -- 172.21.15.114:0/3105165838 wait complete. 2024-03-29T10:43:41.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.859+0000 7fa7a7fac640 10 monclient: build_initial_monmap 2024-03-29T10:43:41.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.859+0000 7fa7a7fac640 10 monclient: monmap: 2024-03-29T10:43:41.844 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:41.844 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:41.844 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:43:41.844 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.859+0000 7fa7a7fac640 1 librados: starting msgr at 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.859+0000 7fa7a7fac640 1 librados: starting objecter 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.859+0000 7fa7a7fac640 1 Processor -- start 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.860+0000 7fa7a7fac640 1 -- start start 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.860+0000 7fa7a7fac640 1 librados: setting wanted keys 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.860+0000 7fa7a7fac640 1 librados: calling monclient init 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.860+0000 7fa7a7fac640 10 monclient: init 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.860+0000 7fa7a7fac640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:41.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.860+0000 7fa7a7fac640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:41.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.864+0000 7fa7a7fac640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a006d9d0 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:41.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.864+0000 7fa7a7fac640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a00b07d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:41.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.864+0000 7fa7a5d48640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fa7a006d9d0 0x7fa7a01d3720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:53751/0 (socket says 172.21.15.114:53751) 2024-03-29T10:43:41.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.864+0000 7fa7a5d48640 1 -- 172.21.15.114:0/2052490766 learned_addr learned my addr 172.21.15.114:0/2052490766 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:41.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.864+0000 7fa7a7fac640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a013ef30 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:41.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.864+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a01d2fb0 con 0x7fa7a013ef30 2024-03-29T10:43:41.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.864+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a01d3290 con 0x7fa7a006d9d0 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.864+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a0133150 con 0x7fa7a00b07d0 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.864+0000 7fa7a7fac640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.864+0000 7fa7a7fac640 10 monclient(hunting): authenticate will time out at 1087.104984s 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.865+0000 7fa7967fc640 10 client.?.objecter ms_handle_connect 0x7fa7a006d9d0 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.865+0000 7fa7967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.865+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165889921 0 0) 0x7fa7a01d3290 con 0x7fa7a006d9d0 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.865+0000 7fa7967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.865+0000 7fa7967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.865+0000 7fa7967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.865+0000 7fa7967fc640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.865+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa784004eb0 con 0x7fa7a006d9d0 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.865+0000 7fa7967fc640 10 client.?.objecter ms_handle_connect 0x7fa7a00b07d0 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.865+0000 7fa7967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:41.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.866+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3873550140 0 0) 0x7fa784004eb0 con 0x7fa7a006d9d0 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.866+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7a01d3290 con 0x7fa7a006d9d0 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.866+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa790003790 con 0x7fa7a006d9d0 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.866+0000 7fa7967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.866+0000 7fa7967fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.866+0000 7fa7967fc640 10 monclient(hunting): dump: 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:41.853 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.866+0000 7fa7967fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.866+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971834477 0 0) 0x7fa7a0133150 con 0x7fa7a00b07d0 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 10 monclient(hunting): my global_id is 4421 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa784004eb0 con 0x7fa7a00b07d0 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2920600345 0 0) 0x7fa7a01d3290 con 0x7fa7a006d9d0 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 >> v1:172.21.15.114:6790/0 conn(0x7fa7a00b07d0 legacy=0x7fa7a0132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 >> v1:172.21.15.114:6789/0 conn(0x7fa7a013ef30 legacy=0x7fa7a0132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 1 monclient: found mon.b 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:41.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a01d3e30 con 0x7fa7a006d9d0 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7a7fac640 5 monclient: authenticate success, global_id 4375 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7a7fac640 10 monclient: _renew_subs 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7a7fac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.867+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a00b33d0 con 0x7fa7a006d9d0 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa7a7fac640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa7a7fac640 10 monclient: _renew_subs 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa7a7fac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a006df90 con 0x7fa7a006d9d0 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa7a7fac640 1 librados: init done 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa790001990 con 0x7fa7a006d9d0 2024-03-29T10:43:41.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa7967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa790003e60 con 0x7fa7a006d9d0 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa797fff640 4 set_mon_vals no callback set 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa7967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa7967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.868+0000 7fa7967fc640 10 monclient: dump: 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:41.856 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:41.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.870+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fa790003fc0 con 0x7fa7a006d9d0 2024-03-29T10:43:41.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.870+0000 7fa7a7fac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:41.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.870+0000 7fa7a7fac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:41.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.870+0000 7fa7a7fac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:41.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.870+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa788005080 con 0x7fa7a006d9d0 2024-03-29T10:43:41.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.871+0000 7fa7a5547640 10 monclient: get_auth_request con 0x7fa78406ddc0 auth_method 0 2024-03-29T10:43:41.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.872+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fa790084ba0 con 0x7fa7a006d9d0 2024-03-29T10:43:41.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.872+0000 7fa7967fc640 10 client.4375.objecter ms_dispatch 0x7fa7a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:41.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.872+0000 7fa7967fc640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:41.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.872+0000 7fa7967fc640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:41.859 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.872+0000 7fa7967fc640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T10:43:41.859 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.872+0000 7fa7967fc640 10 client.4375.objecter ms_handle_connect 0x7fa78406ddc0 2024-03-29T10:43:41.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.880+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa790050480 con 0x7fa7a006d9d0 2024-03-29T10:43:41.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.880+0000 7fa7967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:41.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:41.880+0000 7fa7967fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:42.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.213+0000 7fa7a7fac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:42.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.213+0000 7fa7a7fac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:42.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.213+0000 7fa7a7fac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:42.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.213+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa788005330 con 0x7fa7a006d9d0 2024-03-29T10:43:42.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.215+0000 7fa7967fc640 1 -- 172.21.15.114:0/2052490766 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fa7900523b0 con 0x7fa7a006d9d0 2024-03-29T10:43:42.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.215+0000 7fa7967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:42.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.215+0000 7fa7967fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:42.199 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:43:42.199 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.220+0000 7fa7a7fac640 10 librados: watch_flush enter 2024-03-29T10:43:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.221+0000 7fa7a7fac640 10 librados: watch_flush exit 2024-03-29T10:43:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.221+0000 7fa7a7fac640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T10:43:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.221+0000 7fa7a7fac640 10 client.4375.objecter successfully canceled tick 2024-03-29T10:43:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.221+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 >> v1:172.21.15.114:6816/775044307 conn(0x7fa78406ddc0 legacy=0x7fa784070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.221+0000 7fa7a7fac640 10 monclient: shutdown 2024-03-29T10:43:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.221+0000 7fa7a7fac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.221+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 >> v1:172.21.15.176:6789/0 conn(0x7fa7a006d9d0 legacy=0x7fa7a01d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.222+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 shutdown_connections 2024-03-29T10:43:42.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.222+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 >> 172.21.15.114:0/2052490766 conn(0x7fa7a006b660 msgr2=0x7fa7a00ac6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:42.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.224+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 shutdown_connections 2024-03-29T10:43:42.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.224+0000 7fa7a7fac640 1 -- 172.21.15.114:0/2052490766 wait complete. 2024-03-29T10:43:42.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:42.224+0000 7fa7a7fac640 1 librados: shutdown 2024-03-29T10:43:42.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:49.031 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.199+0000 7fc3cfcfb640 10 monclient: get_monmap_and_config 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.199+0000 7fc3cfcfb640 10 monclient: build_initial_monmap 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.199+0000 7fc3cfcfb640 10 monclient: monmap: 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:43:49.201057+0000 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:43:49.201057+0000 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:43:49.184 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:49.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.200+0000 7fc3cfcfb640 1 Processor -- start 2024-03-29T10:43:49.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.200+0000 7fc3cfcfb640 1 -- start start 2024-03-29T10:43:49.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.200+0000 7fc3cfcfb640 10 monclient: init 2024-03-29T10:43:49.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.201+0000 7fc3cfcfb640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.201+0000 7fc3cfcfb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.201+0000 7fc3cfcfb640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c80b4100 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.201+0000 7fc3cfcfb640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c813ee90 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.201+0000 7fc3cfcfb640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c80af900 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.201+0000 7fc3cfcfb640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c80afd10 con 0x7fc3c813ee90 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.201+0000 7fc3cfcfb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c80af6f0 con 0x7fc3c80af900 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.201+0000 7fc3cfcfb640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c80aaf60 con 0x7fc3c80b4100 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.201+0000 7fc3cfcfb640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.201+0000 7fc3cfcfb640 10 monclient(hunting): authenticate will time out at 1094.442895s 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cd296640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc3c813ee90 0x7fc3c80adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46656/0 (socket says 172.21.15.114:46656) 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cd296640 1 -- 172.21.15.114:0/4156061071 learned_addr learned my addr 172.21.15.114:0/4156061071 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:49.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 1 -- 172.21.15.114:0/4156061071 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29350344 0 0) 0x7fc3c80af6f0 con 0x7fc3c80af900 2024-03-29T10:43:49.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 10 monclient(hunting): _negotiate 2024-03-29T10:43:49.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:49.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:49.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 10 monclient(hunting): my global_id is 4378 2024-03-29T10:43:49.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 1 -- 172.21.15.114:0/4156061071 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b4003520 con 0x7fc3c80af900 2024-03-29T10:43:49.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 1 -- 172.21.15.114:0/4156061071 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711973537 0 0) 0x7fc3c80afd10 con 0x7fc3c813ee90 2024-03-29T10:43:49.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 10 monclient(hunting): _negotiate 2024-03-29T10:43:49.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:49.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:49.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 10 monclient(hunting): my global_id is 4401 2024-03-29T10:43:49.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.202+0000 7fc3cca95640 1 -- 172.21.15.114:0/4156061071 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c80af6f0 con 0x7fc3c813ee90 2024-03-29T10:43:49.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 1 -- 172.21.15.114:0/4156061071 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2440382553 0 0) 0x7fc3c80af6f0 con 0x7fc3c813ee90 2024-03-29T10:43:49.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 1 -- 172.21.15.114:0/4156061071 >> v1:172.21.15.114:6790/0 conn(0x7fc3c80b4100 legacy=0x7fc3c80a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:49.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 1 -- 172.21.15.114:0/4156061071 >> v1:172.21.15.176:6789/0 conn(0x7fc3c80af900 legacy=0x7fc3c813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:49.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 10 monclient: _finish_hunting 0 2024-03-29T10:43:49.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 1 monclient: found mon.noname-a 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 1 -- 172.21.15.114:0/4156061071 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c813e970 con 0x7fc3c813ee90 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 10 monclient: _finish_auth 0 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 10 monclient: _check_auth_tickets 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cfcfb640 5 monclient: authenticate success, global_id 4401 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cfcfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 1 -- 172.21.15.114:0/4156061071 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc3c40015f0 con 0x7fc3c813ee90 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 10 monclient: dump: 2024-03-29T10:43:49.190 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cfcfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 1 -- 172.21.15.114:0/4156061071 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3c4001810 con 0x7fc3c813ee90 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cca95640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cfcfb640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.203+0000 7fc3cfcfb640 4 set_mon_vals no callback set 2024-03-29T10:43:49.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.204+0000 7fc3cca95640 1 -- 172.21.15.114:0/4156061071 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc3c4001ba0 con 0x7fc3c813ee90 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.204+0000 7fc3cca95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.204+0000 7fc3cca95640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.204+0000 7fc3cca95640 10 monclient: dump: 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:49.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.204+0000 7fc3cfcfb640 10 monclient: shutdown 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.204+0000 7fc3cfcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.204+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/4156061071 >> v1:172.21.15.114:6789/0 conn(0x7fc3c813ee90 legacy=0x7fc3c80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.204+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/4156061071 shutdown_connections 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.204+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/4156061071 >> 172.21.15.114:0/4156061071 conn(0x7fc3c80b35e0 msgr2=0x7fc3c80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.205+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/4156061071 shutdown_connections 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.205+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/4156061071 wait complete. 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.205+0000 7fc3cfcfb640 10 monclient: build_initial_monmap 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.205+0000 7fc3cfcfb640 10 monclient: monmap: 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:49.193 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:49.194 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:43:49.194 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:43:49.194 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:49.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.206+0000 7fc3cfcfb640 1 librados: starting msgr at 2024-03-29T10:43:49.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.206+0000 7fc3cfcfb640 1 librados: starting objecter 2024-03-29T10:43:49.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.206+0000 7fc3cfcfb640 1 Processor -- start 2024-03-29T10:43:49.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.206+0000 7fc3cfcfb640 1 -- start start 2024-03-29T10:43:49.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.206+0000 7fc3cfcfb640 1 librados: setting wanted keys 2024-03-29T10:43:49.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.206+0000 7fc3cfcfb640 1 librados: calling monclient init 2024-03-29T10:43:49.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.206+0000 7fc3cfcfb640 10 monclient: init 2024-03-29T10:43:49.198 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cfcfb640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:49.198 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cfcfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:49.198 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cfcfb640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c80af900 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:49.198 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cfcfb640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c80b4100 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cfcfb640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c813ee90 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cd296640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fc3c80b4100 0x7fc3c806e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:48220/0 (socket says 172.21.15.114:48220) 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cd296640 1 -- 172.21.15.114:0/2885126962 learned_addr learned my addr 172.21.15.114:0/2885126962 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cfcfb640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c8132b00 con 0x7fc3c80af900 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c8132de0 con 0x7fc3c80b4100 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3be7fc640 10 client.?.objecter ms_handle_connect 0x7fc3c80b4100 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c81330c0 con 0x7fc3c813ee90 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cfcfb640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3cfcfb640 10 monclient(hunting): authenticate will time out at 1094.448723s 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3be7fc640 10 client.?.objecter ms_handle_connect 0x7fc3c80af900 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3be7fc640 10 client.?.objecter ms_handle_connect 0x7fc3c813ee90 2024-03-29T10:43:49.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.207+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287865028 0 0) 0x7fc3c8132de0 con 0x7fc3c80b4100 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): my global_id is 4381 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac004eb0 con 0x7fc3c80b4100 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305258062 0 0) 0x7fc3c8132b00 con 0x7fc3c80af900 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): my global_id is 4404 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c8132de0 con 0x7fc3c80af900 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960895884 0 0) 0x7fc3c81330c0 con 0x7fc3c813ee90 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:49.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c8132b00 con 0x7fc3c813ee90 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3068052532 0 0) 0x7fc3ac004eb0 con 0x7fc3c80b4100 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3c81330c0 con 0x7fc3c80b4100 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc3c4001b20 con 0x7fc3c80b4100 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): dump: 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:49.201 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.208+0000 7fc3be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4283788431 0 0) 0x7fc3c8132b00 con 0x7fc3c813ee90 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3ac004eb0 con 0x7fc3c813ee90 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc3c0001b80 con 0x7fc3c813ee90 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 10 monclient(hunting): dump: 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:49.202 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4273204210 0 0) 0x7fc3c81330c0 con 0x7fc3c80b4100 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 >> v1:172.21.15.114:6790/0 conn(0x7fc3c813ee90 legacy=0x7fc3c806f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 >> v1:172.21.15.114:6789/0 conn(0x7fc3c80af900 legacy=0x7fc3c80717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 1 monclient: found mon.b 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c80722e0 con 0x7fc3c80b4100 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:49.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3cfcfb640 5 monclient: authenticate success, global_id 4381 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3cfcfb640 10 monclient: _renew_subs 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3cfcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c8071ee0 con 0x7fc3c80b4100 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3cfcfb640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3cfcfb640 10 monclient: _renew_subs 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3cfcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c81e3650 con 0x7fc3c80b4100 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.209+0000 7fc3cfcfb640 1 librados: init done 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.211+0000 7fc3cfcfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.211+0000 7fc3cfcfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.211+0000 7fc3cfcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.211+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3c4001c80 con 0x7fc3c80b4100 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.211+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c806dfd0 con 0x7fc3c80b4100 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.212+0000 7fc3be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:49.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.212+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc3c4004c30 con 0x7fc3c80b4100 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.212+0000 7fc3be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.212+0000 7fc3be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.212+0000 7fc3be7fc640 10 monclient: dump: 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.212+0000 7fc3bffff640 4 set_mon_vals no callback set 2024-03-29T10:43:49.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.212+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fc3c401ac00 con 0x7fc3c80b4100 2024-03-29T10:43:49.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.213+0000 7fc3cda97640 10 monclient: get_auth_request con 0x7fc3ac06de00 auth_method 0 2024-03-29T10:43:49.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.214+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fc3c4084a10 con 0x7fc3c80b4100 2024-03-29T10:43:49.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.214+0000 7fc3be7fc640 10 client.4381.objecter ms_dispatch 0x7fc3c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:49.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.214+0000 7fc3be7fc640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:49.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.214+0000 7fc3be7fc640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:49.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.214+0000 7fc3be7fc640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-29T10:43:49.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.214+0000 7fc3be7fc640 10 client.4381.objecter ms_handle_connect 0x7fc3ac06de00 2024-03-29T10:43:49.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.221+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3c40503a0 con 0x7fc3c80b4100 2024-03-29T10:43:49.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.221+0000 7fc3be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:49.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.221+0000 7fc3be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:49.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.572+0000 7fc3cfcfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:49.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.572+0000 7fc3cfcfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:49.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.572+0000 7fc3cfcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:43:49.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.572+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3c8075f10 con 0x7fc3c80b4100 2024-03-29T10:43:49.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.573+0000 7fc3be7fc640 1 -- 172.21.15.114:0/2885126962 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fc3c40522d0 con 0x7fc3c80b4100 2024-03-29T10:43:49.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.573+0000 7fc3be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:49.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.573+0000 7fc3be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:49.557 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:43:49.557 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.577+0000 7fc3cfcfb640 10 librados: watch_flush enter 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.577+0000 7fc3cfcfb640 10 librados: watch_flush exit 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.577+0000 7fc3cfcfb640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.577+0000 7fc3cfcfb640 10 client.4381.objecter successfully canceled tick 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.578+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 >> v1:172.21.15.114:6816/775044307 conn(0x7fc3ac06de00 legacy=0x7fc3ac0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.578+0000 7fc3cfcfb640 10 monclient: shutdown 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.578+0000 7fc3cfcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.578+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 >> v1:172.21.15.176:6789/0 conn(0x7fc3c80b4100 legacy=0x7fc3c806e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.578+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 shutdown_connections 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.578+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 >> 172.21.15.114:0/2885126962 conn(0x7fc3c80b35e0 msgr2=0x7fc3c80b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.578+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 shutdown_connections 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.578+0000 7fc3cfcfb640 1 -- 172.21.15.114:0/2885126962 wait complete. 2024-03-29T10:43:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:49.578+0000 7fc3cfcfb640 1 librados: shutdown 2024-03-29T10:43:49.578 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:56.385 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:56.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.553+0000 7f981a489640 10 monclient: get_monmap_and_config 2024-03-29T10:43:56.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.553+0000 7f981a489640 10 monclient: build_initial_monmap 2024-03-29T10:43:56.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.553+0000 7f981a489640 10 monclient: monmap: 2024-03-29T10:43:56.537 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:56.538 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:56.538 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:43:56.555203+0000 2024-03-29T10:43:56.538 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:43:56.555203+0000 2024-03-29T10:43:56.538 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:56.538 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:56.538 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:56.538 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:43:56.538 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:43:56.538 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:56.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.556+0000 7f981a489640 1 Processor -- start 2024-03-29T10:43:56.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.556+0000 7f981a489640 1 -- start start 2024-03-29T10:43:56.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.556+0000 7f981a489640 10 monclient: init 2024-03-29T10:43:56.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.557+0000 7f981a489640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:56.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.557+0000 7f981a489640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:56.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.557+0000 7f981a489640 10 monclient(hunting): picked mon.noname-a con 0x7f980c006a70 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:56.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.557+0000 7f981a489640 10 monclient(hunting): picked mon.noname-c con 0x7f980c02b100 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:56.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.557+0000 7f981a489640 10 monclient(hunting): picked mon.noname-b con 0x7f980c029c30 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:56.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.557+0000 7f981a489640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c02a270 con 0x7f980c006a70 2024-03-29T10:43:56.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.557+0000 7f981a489640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c02c440 con 0x7f980c02b100 2024-03-29T10:43:56.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.557+0000 7f981a489640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c0c9440 con 0x7f980c029c30 2024-03-29T10:43:56.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.557+0000 7f981a489640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:56.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.557+0000 7f981a489640 10 monclient(hunting): authenticate will time out at 1101.798482s 2024-03-29T10:43:56.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.560+0000 7f9818c86640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f980c02b100 0x7f980c0cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:34935/0 (socket says 172.21.15.114:34935) 2024-03-29T10:43:56.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.561+0000 7f9818c86640 1 -- 172.21.15.114:0/1875887112 learned_addr learned my addr 172.21.15.114:0/1875887112 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:56.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.563+0000 7f980bfff640 1 -- 172.21.15.114:0/1875887112 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373355539 0 0) 0x7f980c02a270 con 0x7f980c006a70 2024-03-29T10:43:56.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.563+0000 7f980bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.563+0000 7f980bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.563+0000 7f980bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.563+0000 7f980bfff640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.563+0000 7f980bfff640 1 -- 172.21.15.114:0/1875887112 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9800003520 con 0x7f980c006a70 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 1 -- 172.21.15.114:0/1875887112 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422428808 0 0) 0x7f980c02c440 con 0x7f980c02b100 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 1 -- 172.21.15.114:0/1875887112 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980c02a270 con 0x7f980c02b100 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 1 -- 172.21.15.114:0/1875887112 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3649371372 0 0) 0x7f9800003520 con 0x7f980c006a70 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 1 -- 172.21.15.114:0/1875887112 >> v1:172.21.15.114:6790/0 conn(0x7f980c029c30 legacy=0x7f980c0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:56.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 1 -- 172.21.15.114:0/1875887112 >> v1:172.21.15.176:6789/0 conn(0x7f980c02b100 legacy=0x7f980c0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 1 monclient: found mon.noname-a 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 1 -- 172.21.15.114:0/1875887112 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c0c95d0 con 0x7f980c006a70 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient: _finish_auth 0 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 1 -- 172.21.15.114:0/1875887112 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9804001690 con 0x7f980c006a70 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient: dump: 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:56.551 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f981a489640 5 monclient: authenticate success, global_id 4407 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f981a489640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 1 -- 172.21.15.114:0/1875887112 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9804001860 con 0x7f980c006a70 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 1 -- 172.21.15.114:0/1875887112 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9804001cb0 con 0x7f980c006a70 2024-03-29T10:43:56.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f980bfff640 10 monclient: dump: 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f981a489640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:56.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.564+0000 7f981a489640 4 set_mon_vals no callback set 2024-03-29T10:43:56.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.565+0000 7f981a489640 10 monclient: shutdown 2024-03-29T10:43:56.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.565+0000 7f981a489640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:56.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.565+0000 7f981a489640 1 -- 172.21.15.114:0/1875887112 >> v1:172.21.15.114:6789/0 conn(0x7f980c006a70 legacy=0x7f980c0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:56.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.569+0000 7f981a489640 1 -- 172.21.15.114:0/1875887112 shutdown_connections 2024-03-29T10:43:56.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.569+0000 7f981a489640 1 -- 172.21.15.114:0/1875887112 >> 172.21.15.114:0/1875887112 conn(0x7f980c0058f0 msgr2=0x7f980c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:56.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.569+0000 7f981a489640 1 -- 172.21.15.114:0/1875887112 shutdown_connections 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.569+0000 7f981a489640 1 -- 172.21.15.114:0/1875887112 wait complete. 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.570+0000 7f981a489640 10 monclient: build_initial_monmap 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.570+0000 7f981a489640 10 monclient: monmap: 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:56.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.571+0000 7f981a489640 1 librados: starting msgr at 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.571+0000 7f981a489640 1 librados: starting objecter 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.571+0000 7f981a489640 1 Processor -- start 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.571+0000 7f981a489640 1 -- start start 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.571+0000 7f981a489640 1 librados: setting wanted keys 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.571+0000 7f981a489640 1 librados: calling monclient init 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.571+0000 7f981a489640 10 monclient: init 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.572+0000 7f981a489640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.572+0000 7f981a489640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.572+0000 7f981a489640 10 monclient(hunting): picked mon.noname-b con 0x7f980c006a70 addr v1:172.21.15.176:6789/0 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.572+0000 7f981a489640 10 monclient(hunting): picked mon.noname-a con 0x7f980c029c30 addr v1:172.21.15.114:6789/0 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.572+0000 7f981a489640 10 monclient(hunting): picked mon.noname-c con 0x7f980c164db0 addr v1:172.21.15.114:6790/0 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.572+0000 7f981a489640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c02f260 con 0x7f980c029c30 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.572+0000 7f981a489640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c02f540 con 0x7f980c006a70 2024-03-29T10:43:56.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.572+0000 7f981a489640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c02d250 con 0x7f980c164db0 2024-03-29T10:43:56.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.572+0000 7f981a489640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:56.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.572+0000 7f981a489640 10 monclient(hunting): authenticate will time out at 1101.813311s 2024-03-29T10:43:56.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.578+0000 7f9819487640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f980c006a70 0x7f980c031dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:35029/0 (socket says 172.21.15.114:35029) 2024-03-29T10:43:56.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.578+0000 7f9819487640 1 -- 172.21.15.114:0/2476176936 learned_addr learned my addr 172.21.15.114:0/2476176936 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:43:56.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.578+0000 7f9819c88640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f980c164db0 0x7f980c16c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:60577/0 (socket says 172.21.15.114:60577) 2024-03-29T10:43:56.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.578+0000 7f9809ffb640 10 client.?.objecter ms_handle_connect 0x7f980c164db0 2024-03-29T10:43:56.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.578+0000 7f9809ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:56.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.578+0000 7f9809ffb640 10 client.?.objecter ms_handle_connect 0x7f980c006a70 2024-03-29T10:43:56.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.578+0000 7f9809ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:56.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.579+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240225710 0 0) 0x7f980c02d250 con 0x7f980c164db0 2024-03-29T10:43:56.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.579+0000 7f9809ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:56.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.579+0000 7f9809ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:56.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.579+0000 7f9809ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:56.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.579+0000 7f9809ffb640 10 monclient(hunting): my global_id is 4433 2024-03-29T10:43:56.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.579+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f8004eb0 con 0x7f980c164db0 2024-03-29T10:43:56.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.582+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941618056 0 0) 0x7f980c02f540 con 0x7f980c006a70 2024-03-29T10:43:56.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.583+0000 7f9809ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:56.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.583+0000 7f9809ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:56.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.583+0000 7f9809ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:56.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.583+0000 7f9809ffb640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:43:56.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.583+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980c02d250 con 0x7f980c006a70 2024-03-29T10:43:56.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.584+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235502792 0 0) 0x7f97f8004eb0 con 0x7f980c164db0 2024-03-29T10:43:56.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.584+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f980c02f540 con 0x7f980c164db0 2024-03-29T10:43:56.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.584+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f98100018b0 con 0x7f980c164db0 2024-03-29T10:43:56.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.584+0000 7f9809ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:56.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.584+0000 7f9809ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:56.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.584+0000 7f9809ffb640 10 monclient(hunting): dump: 2024-03-29T10:43:56.567 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:56.567 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:56.567 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:56.567 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:56.567 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:56.568 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:56.568 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:56.568 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:56.568 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:56.568 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:56.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.584+0000 7f9809ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:56.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 10 client.?.objecter ms_handle_connect 0x7f980c029c30 2024-03-29T10:43:56.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:56.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1383302663 0 0) 0x7f980c02f540 con 0x7f980c164db0 2024-03-29T10:43:56.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 >> v1:172.21.15.176:6789/0 conn(0x7f980c006a70 legacy=0x7f980c031dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:56.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 >> v1:172.21.15.114:6789/0 conn(0x7f980c029c30 legacy=0x7f980c02d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:56.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:43:56.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 1 monclient: found mon.c 2024-03-29T10:43:56.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:56.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:56.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c02cee0 con 0x7f980c164db0 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 10 monclient: _finish_auth 0 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f9809ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f981a489640 5 monclient: authenticate success, global_id 4433 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f981a489640 10 monclient: _renew_subs 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f981a489640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.585+0000 7f981a489640 1 -- 172.21.15.114:0/2476176936 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c0324e0 con 0x7f980c164db0 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.586+0000 7f981a489640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.586+0000 7f981a489640 10 monclient: _renew_subs 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.586+0000 7f981a489640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.586+0000 7f981a489640 1 -- 172.21.15.114:0/2476176936 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c164550 con 0x7f980c164db0 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.586+0000 7f981a489640 1 librados: init done 2024-03-29T10:43:56.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.586+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9810001d60 con 0x7f980c164db0 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.586+0000 7f9809ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.586+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f98100037f0 con 0x7f980c164db0 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.586+0000 7f9809ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.586+0000 7f9809ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.586+0000 7f9809ffb640 10 monclient: dump: 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:43:56.574 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:43:56.575 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:43:56.575 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:43:56.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.587+0000 7f980affd640 4 set_mon_vals no callback set 2024-03-29T10:43:56.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.589+0000 7f981a489640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:56.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.589+0000 7f981a489640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:56.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.589+0000 7f981a489640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:56.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.590+0000 7f981a489640 1 -- 172.21.15.114:0/2476176936 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f4005080 con 0x7f980c164db0 2024-03-29T10:43:56.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.591+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f9810003950 con 0x7f980c164db0 2024-03-29T10:43:56.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.591+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f9810084980 con 0x7f980c164db0 2024-03-29T10:43:56.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.591+0000 7f9809ffb640 10 client.4433.objecter ms_dispatch 0x7f9814000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:56.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.591+0000 7f9809ffb640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:56.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.591+0000 7f9809ffb640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:56.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.591+0000 7f9809ffb640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-29T10:43:56.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.594+0000 7f9819487640 10 monclient: get_auth_request con 0x7f97f806db70 auth_method 0 2024-03-29T10:43:56.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.595+0000 7f9809ffb640 10 client.4433.objecter ms_handle_connect 0x7f97f806db70 2024-03-29T10:43:56.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.600+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9810050090 con 0x7f980c164db0 2024-03-29T10:43:56.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.600+0000 7f9809ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:56.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.600+0000 7f9809ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:56.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.921+0000 7f981a489640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:56.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.921+0000 7f981a489640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:56.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.921+0000 7f981a489640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:43:56.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.921+0000 7f981a489640 1 -- 172.21.15.114:0/2476176936 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97f4005330 con 0x7f980c164db0 2024-03-29T10:43:56.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.923+0000 7f9809ffb640 1 -- 172.21.15.114:0/2476176936 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f9810052190 con 0x7f980c164db0 2024-03-29T10:43:56.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.923+0000 7f9809ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:56.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.923+0000 7f9809ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:56.906 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:43:56.907 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:56.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.926+0000 7f981a489640 10 librados: watch_flush enter 2024-03-29T10:43:56.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.927+0000 7f981a489640 10 librados: watch_flush exit 2024-03-29T10:43:56.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.927+0000 7f981a489640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-29T10:43:56.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.927+0000 7f981a489640 10 client.4433.objecter successfully canceled tick 2024-03-29T10:43:56.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.927+0000 7f981a489640 1 -- 172.21.15.114:0/2476176936 >> v1:172.21.15.114:6816/775044307 conn(0x7f97f806db70 legacy=0x7f97f8070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:56.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.927+0000 7f981a489640 10 monclient: shutdown 2024-03-29T10:43:56.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.927+0000 7f981a489640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:56.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.927+0000 7f981a489640 1 -- 172.21.15.114:0/2476176936 >> v1:172.21.15.114:6790/0 conn(0x7f980c164db0 legacy=0x7f980c16c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:56.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.928+0000 7f981a489640 1 -- 172.21.15.114:0/2476176936 shutdown_connections 2024-03-29T10:43:56.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.928+0000 7f981a489640 1 -- 172.21.15.114:0/2476176936 >> 172.21.15.114:0/2476176936 conn(0x7f980c0058f0 msgr2=0x7f980c0c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:56.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.928+0000 7f981a489640 1 -- 172.21.15.114:0/2476176936 shutdown_connections 2024-03-29T10:43:56.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.928+0000 7f981a489640 1 -- 172.21.15.114:0/2476176936 wait complete. 2024-03-29T10:43:56.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:43:56.928+0000 7f981a489640 1 librados: shutdown 2024-03-29T10:43:56.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:03.734 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:03.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.902+0000 7fc0ecad0640 10 monclient: get_monmap_and_config 2024-03-29T10:44:03.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.902+0000 7fc0ecad0640 10 monclient: build_initial_monmap 2024-03-29T10:44:03.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.902+0000 7fc0ecad0640 10 monclient: monmap: 2024-03-29T10:44:03.886 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:03.886 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:03.887 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:44:03.904159+0000 2024-03-29T10:44:03.887 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:44:03.904159+0000 2024-03-29T10:44:03.887 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:03.887 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:03.887 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:03.887 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:44:03.887 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:44:03.887 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:03.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.904+0000 7fc0ecad0640 1 Processor -- start 2024-03-29T10:44:03.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.904+0000 7fc0ecad0640 1 -- start start 2024-03-29T10:44:03.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.904+0000 7fc0ecad0640 10 monclient: init 2024-03-29T10:44:03.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0ecad0640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:03.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0ecad0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:03.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0ecad0640 10 monclient(hunting): picked mon.noname-b con 0x7fc0e81457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:03.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0ecad0640 10 monclient(hunting): picked mon.noname-c con 0x7fc0e813f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:03.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0ecad0640 10 monclient(hunting): picked mon.noname-a con 0x7fc0e80a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:03.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0ecad0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e80aaef0 con 0x7fc0e80a9d70 2024-03-29T10:44:03.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0ecad0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e8145bd0 con 0x7fc0e813f650 2024-03-29T10:44:03.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0ecad0640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e8145eb0 con 0x7fc0e81457f0 2024-03-29T10:44:03.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0ecad0640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:03.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0ecad0640 10 monclient(hunting): authenticate will time out at 1109.146587s 2024-03-29T10:44:03.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0e659c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fc0e81457f0 0x7fc0e813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:53544/0 (socket says 172.21.15.114:53544) 2024-03-29T10:44:03.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.905+0000 7fc0e659c640 1 -- 172.21.15.114:0/1025661431 learned_addr learned my addr 172.21.15.114:0/1025661431 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:03.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.906+0000 7fc0e559a640 1 -- 172.21.15.114:0/1025661431 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307736589 0 0) 0x7fc0e8145eb0 con 0x7fc0e81457f0 2024-03-29T10:44:03.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.906+0000 7fc0e559a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:03.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.906+0000 7fc0e559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:03.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.906+0000 7fc0e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:03.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.906+0000 7fc0e559a640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:44:03.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.906+0000 7fc0e559a640 1 -- 172.21.15.114:0/1025661431 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d0003520 con 0x7fc0e81457f0 2024-03-29T10:44:03.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.907+0000 7fc0e559a640 1 -- 172.21.15.114:0/1025661431 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2939721594 0 0) 0x7fc0d0003520 con 0x7fc0e81457f0 2024-03-29T10:44:03.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.907+0000 7fc0e559a640 1 -- 172.21.15.114:0/1025661431 >> v1:172.21.15.176:6789/0 conn(0x7fc0e813f650 legacy=0x7fc0e80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.909+0000 7fc0e559a640 1 -- 172.21.15.114:0/1025661431 >> v1:172.21.15.114:6789/0 conn(0x7fc0e80a9d70 legacy=0x7fc0e814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 10 monclient: _finish_hunting 0 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 1 monclient: found mon.noname-b 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 1 -- 172.21.15.114:0/1025661431 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e8146340 con 0x7fc0e81457f0 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 10 monclient: _finish_auth 0 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 10 monclient: _check_auth_tickets 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 1 -- 172.21.15.114:0/1025661431 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc0d8001690 con 0x7fc0e81457f0 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.910+0000 7fc0e559a640 10 monclient: dump: 2024-03-29T10:44:03.894 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:03.895 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:03.895 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:03.895 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:03.895 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:03.895 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:03.895 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:03.895 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:03.895 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:03.895 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:03.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.913+0000 7fc0ecad0640 5 monclient: authenticate success, global_id 4436 2024-03-29T10:44:03.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.913+0000 7fc0ecad0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:03.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.913+0000 7fc0e559a640 1 -- 172.21.15.114:0/1025661431 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0d8001860 con 0x7fc0e81457f0 2024-03-29T10:44:03.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.913+0000 7fc0e559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:03.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.913+0000 7fc0e559a640 1 -- 172.21.15.114:0/1025661431 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc0d8001cb0 con 0x7fc0e81457f0 2024-03-29T10:44:03.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.913+0000 7fc0e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:03.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.913+0000 7fc0e559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:03.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.913+0000 7fc0e559a640 10 monclient: dump: 2024-03-29T10:44:03.897 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:03.897 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:03.898 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:03.898 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:03.898 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:03.898 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:03.898 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:03.898 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:03.898 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:03.898 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:03.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.914+0000 7fc0ecad0640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:03.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.914+0000 7fc0ecad0640 4 set_mon_vals no callback set 2024-03-29T10:44:03.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.915+0000 7fc0ecad0640 10 monclient: shutdown 2024-03-29T10:44:03.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.915+0000 7fc0ecad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:03.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.915+0000 7fc0ecad0640 1 -- 172.21.15.114:0/1025661431 >> v1:172.21.15.114:6790/0 conn(0x7fc0e81457f0 legacy=0x7fc0e813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:03.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.915+0000 7fc0ecad0640 1 -- 172.21.15.114:0/1025661431 shutdown_connections 2024-03-29T10:44:03.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.915+0000 7fc0ecad0640 1 -- 172.21.15.114:0/1025661431 >> 172.21.15.114:0/1025661431 conn(0x7fc0e8066a00 msgr2=0x7fc0e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:03.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.917+0000 7fc0ecad0640 1 -- 172.21.15.114:0/1025661431 shutdown_connections 2024-03-29T10:44:03.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.917+0000 7fc0ecad0640 1 -- 172.21.15.114:0/1025661431 wait complete. 2024-03-29T10:44:03.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.918+0000 7fc0ecad0640 10 monclient: build_initial_monmap 2024-03-29T10:44:03.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.918+0000 7fc0ecad0640 10 monclient: monmap: 2024-03-29T10:44:03.901 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:03.901 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:03.901 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:44:03.901 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:44:03.901 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:03.901 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:03.901 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:03.902 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:44:03.902 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:44:03.902 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:03.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.918+0000 7fc0ecad0640 1 librados: starting msgr at 2024-03-29T10:44:03.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.918+0000 7fc0ecad0640 1 librados: starting objecter 2024-03-29T10:44:03.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.918+0000 7fc0ecad0640 1 Processor -- start 2024-03-29T10:44:03.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.918+0000 7fc0ecad0640 1 -- start start 2024-03-29T10:44:03.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.918+0000 7fc0ecad0640 1 librados: setting wanted keys 2024-03-29T10:44:03.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.918+0000 7fc0ecad0640 1 librados: calling monclient init 2024-03-29T10:44:03.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.918+0000 7fc0ecad0640 10 monclient: init 2024-03-29T10:44:03.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.919+0000 7fc0ecad0640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:03.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.919+0000 7fc0ecad0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:03.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.919+0000 7fc0ecad0640 10 monclient(hunting): picked mon.noname-c con 0x7fc0e80a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:03.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.919+0000 7fc0ecad0640 10 monclient(hunting): picked mon.noname-a con 0x7fc0e813f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:03.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.919+0000 7fc0ecad0640 10 monclient(hunting): picked mon.noname-b con 0x7fc0e81457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:03.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.919+0000 7fc0ecad0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e8136770 con 0x7fc0e813f650 2024-03-29T10:44:03.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.919+0000 7fc0ecad0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e8137270 con 0x7fc0e81457f0 2024-03-29T10:44:03.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.919+0000 7fc0ecad0640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e8137550 con 0x7fc0e80a9d70 2024-03-29T10:44:03.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.919+0000 7fc0ecad0640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:03.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.919+0000 7fc0ecad0640 10 monclient(hunting): authenticate will time out at 1109.160677s 2024-03-29T10:44:03.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.921+0000 7fc0e659c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fc0e80a9d70 0x7fc0e80708c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55764/0 (socket says 172.21.15.114:55764) 2024-03-29T10:44:03.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.921+0000 7fc0e659c640 1 -- 172.21.15.114:0/3221530230 learned_addr learned my addr 172.21.15.114:0/3221530230 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:03.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.921+0000 7fc0d6ffd640 10 client.?.objecter ms_handle_connect 0x7fc0e80a9d70 2024-03-29T10:44:03.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.921+0000 7fc0d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:03.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.921+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611225588 0 0) 0x7fc0e8137550 con 0x7fc0e80a9d70 2024-03-29T10:44:03.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.921+0000 7fc0d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:03.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.921+0000 7fc0d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:03.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.921+0000 7fc0d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:03.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.921+0000 7fc0d6ffd640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:44:03.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.921+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c8004eb0 con 0x7fc0e80a9d70 2024-03-29T10:44:03.907 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2345275723 0 0) 0x7fc0c8004eb0 con 0x7fc0e80a9d70 2024-03-29T10:44:03.907 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0e8137550 con 0x7fc0e80a9d70 2024-03-29T10:44:03.907 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc0d8001c30 con 0x7fc0e80a9d70 2024-03-29T10:44:03.907 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:03.907 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:03.907 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 10 monclient(hunting): dump: 2024-03-29T10:44:03.907 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:03.907 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2724233508 0 0) 0x7fc0e8137550 con 0x7fc0e80a9d70 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 >> v1:172.21.15.176:6789/0 conn(0x7fc0e81457f0 legacy=0x7fc0e81e65c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 >> v1:172.21.15.114:6789/0 conn(0x7fc0e813f650 legacy=0x7fc0e8071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:03.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 1 monclient: found mon.c 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e8137830 con 0x7fc0e80a9d70 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0ecad0640 5 monclient: authenticate success, global_id 4439 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0ecad0640 10 monclient: _renew_subs 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0ecad0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0ecad0640 1 -- 172.21.15.114:0/3221530230 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e8071970 con 0x7fc0e80a9d70 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0ecad0640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0ecad0640 10 monclient: _renew_subs 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0ecad0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0ecad0640 1 -- 172.21.15.114:0/3221530230 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e8135840 con 0x7fc0e80a9d70 2024-03-29T10:44:03.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.922+0000 7fc0ecad0640 1 librados: init done 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.923+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0d8004a90 con 0x7fc0e80a9d70 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.923+0000 7fc0d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.923+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc0d8004ee0 con 0x7fc0e80a9d70 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.923+0000 7fc0d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.923+0000 7fc0d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.923+0000 7fc0d6ffd640 10 monclient: dump: 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:03.910 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:03.911 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:03.911 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:03.911 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:03.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.925+0000 7fc0e4d99640 4 set_mon_vals no callback set 2024-03-29T10:44:03.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.925+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fc0d8005040 con 0x7fc0e80a9d70 2024-03-29T10:44:03.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.926+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fc0d8084c20 con 0x7fc0e80a9d70 2024-03-29T10:44:03.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.926+0000 7fc0d6ffd640 10 client.4439.objecter ms_dispatch 0x7fc0e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:03.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.926+0000 7fc0d6ffd640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:03.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.926+0000 7fc0d6ffd640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:03.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.926+0000 7fc0d6ffd640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T10:44:03.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.926+0000 7fc0ecad0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:03.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.926+0000 7fc0ecad0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:03.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.926+0000 7fc0ecad0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:03.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.926+0000 7fc0ecad0640 1 -- 172.21.15.114:0/3221530230 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0cc005080 con 0x7fc0e80a9d70 2024-03-29T10:44:03.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.926+0000 7fc0e5d9b640 10 monclient: get_auth_request con 0x7fc0c806dc80 auth_method 0 2024-03-29T10:44:03.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.926+0000 7fc0d6ffd640 10 client.4439.objecter ms_handle_connect 0x7fc0c806dc80 2024-03-29T10:44:03.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.934+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0d8050500 con 0x7fc0e80a9d70 2024-03-29T10:44:03.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.934+0000 7fc0d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:03.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:03.934+0000 7fc0d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:04.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.259+0000 7fc0ecad0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:04.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.259+0000 7fc0ecad0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:04.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.259+0000 7fc0ecad0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:04.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.259+0000 7fc0ecad0640 1 -- 172.21.15.114:0/3221530230 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0cc005330 con 0x7fc0e80a9d70 2024-03-29T10:44:04.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.260+0000 7fc0d6ffd640 1 -- 172.21.15.114:0/3221530230 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fc0d8052430 con 0x7fc0e80a9d70 2024-03-29T10:44:04.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.260+0000 7fc0d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:04.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.260+0000 7fc0d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:04.244 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:44:04.244 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:04.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.263+0000 7fc0ecad0640 10 librados: watch_flush enter 2024-03-29T10:44:04.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.263+0000 7fc0ecad0640 10 librados: watch_flush exit 2024-03-29T10:44:04.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.263+0000 7fc0ecad0640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T10:44:04.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.263+0000 7fc0ecad0640 10 client.4439.objecter successfully canceled tick 2024-03-29T10:44:04.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.263+0000 7fc0ecad0640 1 -- 172.21.15.114:0/3221530230 >> v1:172.21.15.114:6816/775044307 conn(0x7fc0c806dc80 legacy=0x7fc0c8070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:04.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.263+0000 7fc0ecad0640 10 monclient: shutdown 2024-03-29T10:44:04.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.263+0000 7fc0ecad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:04.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.263+0000 7fc0ecad0640 1 -- 172.21.15.114:0/3221530230 >> v1:172.21.15.114:6790/0 conn(0x7fc0e80a9d70 legacy=0x7fc0e80708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:04.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.264+0000 7fc0ecad0640 1 -- 172.21.15.114:0/3221530230 shutdown_connections 2024-03-29T10:44:04.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.264+0000 7fc0ecad0640 1 -- 172.21.15.114:0/3221530230 >> 172.21.15.114:0/3221530230 conn(0x7fc0e8066a00 msgr2=0x7fc0e80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:04.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.265+0000 7fc0ecad0640 1 -- 172.21.15.114:0/3221530230 shutdown_connections 2024-03-29T10:44:04.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.265+0000 7fc0ecad0640 1 -- 172.21.15.114:0/3221530230 wait complete. 2024-03-29T10:44:04.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:04.266+0000 7fc0ecad0640 1 librados: shutdown 2024-03-29T10:44:04.266 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:11.072 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:11.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.242+0000 7ff1516a5640 10 monclient: get_monmap_and_config 2024-03-29T10:44:11.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.242+0000 7ff1516a5640 10 monclient: build_initial_monmap 2024-03-29T10:44:11.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.242+0000 7ff1516a5640 10 monclient: monmap: 2024-03-29T10:44:11.225 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:11.225 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:11.226 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:44:11.243400+0000 2024-03-29T10:44:11.226 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:44:11.243400+0000 2024-03-29T10:44:11.226 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:11.226 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:11.226 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:11.226 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:44:11.226 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:44:11.226 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:11.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.245+0000 7ff1516a5640 1 Processor -- start 2024-03-29T10:44:11.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.245+0000 7ff1516a5640 1 -- start start 2024-03-29T10:44:11.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.245+0000 7ff1516a5640 10 monclient: init 2024-03-29T10:44:11.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.246+0000 7ff1516a5640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:11.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.246+0000 7ff1516a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:11.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.246+0000 7ff1516a5640 10 monclient(hunting): picked mon.noname-a con 0x7ff14c1457f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:11.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.246+0000 7ff1516a5640 10 monclient(hunting): picked mon.noname-b con 0x7ff14c13f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:11.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.246+0000 7ff1516a5640 10 monclient(hunting): picked mon.noname-c con 0x7ff14c0a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:11.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.246+0000 7ff1516a5640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0aaef0 con 0x7ff14c1457f0 2024-03-29T10:44:11.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.246+0000 7ff1516a5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c145bd0 con 0x7ff14c0a9d70 2024-03-29T10:44:11.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.246+0000 7ff1516a5640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c145eb0 con 0x7ff14c13f650 2024-03-29T10:44:11.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.246+0000 7ff1516a5640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:11.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.246+0000 7ff1516a5640 10 monclient(hunting): authenticate will time out at 1116.487389s 2024-03-29T10:44:11.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.248+0000 7ff14a7fc640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff14c13f650 0x7ff14c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:54014/0 (socket says 172.21.15.114:54014) 2024-03-29T10:44:11.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.248+0000 7ff14a7fc640 1 -- 172.21.15.114:0/606597891 learned_addr learned my addr 172.21.15.114:0/606597891 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:11.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.251+0000 7ff149ffb640 1 -- 172.21.15.114:0/606597891 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204830537 0 0) 0x7ff14c145eb0 con 0x7ff14c13f650 2024-03-29T10:44:11.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.251+0000 7ff149ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:11.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.251+0000 7ff149ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:11.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.251+0000 7ff149ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:11.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.251+0000 7ff149ffb640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:44:11.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.251+0000 7ff149ffb640 1 -- 172.21.15.114:0/606597891 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff134003520 con 0x7ff14c13f650 2024-03-29T10:44:11.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.251+0000 7ff149ffb640 1 -- 172.21.15.114:0/606597891 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3434430007 0 0) 0x7ff134003520 con 0x7ff14c13f650 2024-03-29T10:44:11.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.251+0000 7ff149ffb640 1 -- 172.21.15.114:0/606597891 >> v1:172.21.15.176:6789/0 conn(0x7ff14c0a9d70 legacy=0x7ff14c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 1 -- 172.21.15.114:0/606597891 >> v1:172.21.15.114:6789/0 conn(0x7ff14c1457f0 legacy=0x7ff14c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 1 monclient: found mon.noname-b 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 1 -- 172.21.15.114:0/606597891 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c146340 con 0x7ff14c13f650 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 1 -- 172.21.15.114:0/606597891 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff140001690 con 0x7ff14c13f650 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:11.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff149ffb640 10 monclient: dump: 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff1516a5640 5 monclient: authenticate success, global_id 4442 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.252+0000 7ff1516a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.253+0000 7ff149ffb640 1 -- 172.21.15.114:0/606597891 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff140001920 con 0x7ff14c13f650 2024-03-29T10:44:11.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.253+0000 7ff149ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.253+0000 7ff149ffb640 1 -- 172.21.15.114:0/606597891 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff140001cb0 con 0x7ff14c13f650 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.253+0000 7ff149ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.253+0000 7ff149ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.253+0000 7ff149ffb640 10 monclient: dump: 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:11.240 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:11.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.253+0000 7ff1516a5640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:11.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.253+0000 7ff1516a5640 4 set_mon_vals no callback set 2024-03-29T10:44:11.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.254+0000 7ff1516a5640 10 monclient: shutdown 2024-03-29T10:44:11.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.254+0000 7ff1516a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:11.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.254+0000 7ff1516a5640 1 -- 172.21.15.114:0/606597891 >> v1:172.21.15.114:6790/0 conn(0x7ff14c13f650 legacy=0x7ff14c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:11.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.254+0000 7ff1516a5640 1 -- 172.21.15.114:0/606597891 shutdown_connections 2024-03-29T10:44:11.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.254+0000 7ff1516a5640 1 -- 172.21.15.114:0/606597891 >> 172.21.15.114:0/606597891 conn(0x7ff14c066a00 msgr2=0x7ff14c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:11.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.254+0000 7ff1516a5640 1 -- 172.21.15.114:0/606597891 shutdown_connections 2024-03-29T10:44:11.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.255+0000 7ff1516a5640 1 -- 172.21.15.114:0/606597891 wait complete. 2024-03-29T10:44:11.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.255+0000 7ff1516a5640 10 monclient: build_initial_monmap 2024-03-29T10:44:11.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.255+0000 7ff1516a5640 10 monclient: monmap: 2024-03-29T10:44:11.242 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.256+0000 7ff1516a5640 1 librados: starting msgr at 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.256+0000 7ff1516a5640 1 librados: starting objecter 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.256+0000 7ff1516a5640 1 Processor -- start 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.256+0000 7ff1516a5640 1 -- start start 2024-03-29T10:44:11.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.256+0000 7ff1516a5640 1 librados: setting wanted keys 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.256+0000 7ff1516a5640 1 librados: calling monclient init 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.256+0000 7ff1516a5640 10 monclient: init 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.257+0000 7ff1516a5640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.257+0000 7ff1516a5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.257+0000 7ff1516a5640 10 monclient(hunting): picked mon.noname-a con 0x7ff14c0a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.257+0000 7ff1516a5640 10 monclient(hunting): picked mon.noname-b con 0x7ff14c1457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.257+0000 7ff1516a5640 10 monclient(hunting): picked mon.noname-c con 0x7ff14c06f8d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.257+0000 7ff1516a5640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c137940 con 0x7ff14c0a9d70 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.257+0000 7ff1516a5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c137e90 con 0x7ff14c1457f0 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.257+0000 7ff1516a5640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c137bb0 con 0x7ff14c06f8d0 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.257+0000 7ff1516a5640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.257+0000 7ff1516a5640 10 monclient(hunting): authenticate will time out at 1116.498406s 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.258+0000 7ff14b7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff14c06f8d0 0x7ff14c1eedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55678/0 (socket says 172.21.15.114:55678) 2024-03-29T10:44:11.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.258+0000 7ff14b7fe640 1 -- 172.21.15.114:0/2777572822 learned_addr learned my addr 172.21.15.114:0/2777572822 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:11.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.258+0000 7ff1337fe640 10 client.?.objecter ms_handle_connect 0x7ff14c06f8d0 2024-03-29T10:44:11.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.258+0000 7ff1337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:11.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297701577 0 0) 0x7ff14c137bb0 con 0x7ff14c06f8d0 2024-03-29T10:44:11.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:11.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:11.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:11.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:44:11.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff138004eb0 con 0x7ff14c06f8d0 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3683237137 0 0) 0x7ff138004eb0 con 0x7ff14c06f8d0 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff138005300 con 0x7ff14c06f8d0 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff13c0017b0 con 0x7ff14c06f8d0 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 10 monclient(hunting): dump: 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:11.247 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2357723527 0 0) 0x7ff138005300 con 0x7ff14c06f8d0 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.259+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 >> v1:172.21.15.176:6789/0 conn(0x7ff14c1457f0 legacy=0x7ff14c133d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 >> v1:172.21.15.114:6789/0 conn(0x7ff14c0a9d70 legacy=0x7ff14c071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1337fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1337fe640 1 monclient: found mon.c 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1337fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c06fed0 con 0x7ff14c06f8d0 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1337fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1337fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:11.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1337fe640 10 client.?.objecter ms_handle_connect 0x7ff14c0a9d70 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1516a5640 5 monclient: authenticate success, global_id 4445 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1516a5640 10 monclient: _renew_subs 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1516a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1516a5640 1 -- 172.21.15.114:0/2777572822 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c071be0 con 0x7ff14c06f8d0 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1516a5640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1516a5640 10 monclient: _renew_subs 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1516a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1516a5640 1 -- 172.21.15.114:0/2777572822 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c072070 con 0x7ff14c06f8d0 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.262+0000 7ff1516a5640 1 librados: init done 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.263+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff13c001d80 con 0x7ff14c06f8d0 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.263+0000 7ff1337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.263+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff13c0017b0 con 0x7ff14c06f8d0 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.263+0000 7ff1337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:11.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.263+0000 7ff1337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.263+0000 7ff1337fe640 10 monclient: dump: 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.263+0000 7ff148ff9640 4 set_mon_vals no callback set 2024-03-29T10:44:11.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.264+0000 7ff1516a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:11.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.264+0000 7ff1516a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:11.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.264+0000 7ff1516a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:11.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.264+0000 7ff1516a5640 1 -- 172.21.15.114:0/2777572822 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12c005080 con 0x7ff14c06f8d0 2024-03-29T10:44:11.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.265+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7ff13c01a770 con 0x7ff14c06f8d0 2024-03-29T10:44:11.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.266+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7ff13c0847c0 con 0x7ff14c06f8d0 2024-03-29T10:44:11.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.266+0000 7ff1337fe640 10 client.4445.objecter ms_dispatch 0x7ff14c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:11.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.266+0000 7ff1337fe640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:11.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.266+0000 7ff1337fe640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:11.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.266+0000 7ff1337fe640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T10:44:11.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.269+0000 7ff14affd640 10 monclient: get_auth_request con 0x7ff13806df70 auth_method 0 2024-03-29T10:44:11.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.269+0000 7ff1337fe640 10 client.4445.objecter ms_handle_connect 0x7ff13806df70 2024-03-29T10:44:11.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.273+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff13c04fed0 con 0x7ff14c06f8d0 2024-03-29T10:44:11.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.273+0000 7ff1337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:11.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.273+0000 7ff1337fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:11.593 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:44:11.593 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:11.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.607+0000 7ff1516a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:11.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.607+0000 7ff1516a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:11.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.607+0000 7ff1516a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:11.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.607+0000 7ff1516a5640 1 -- 172.21.15.114:0/2777572822 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff12c005330 con 0x7ff14c06f8d0 2024-03-29T10:44:11.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.609+0000 7ff1337fe640 1 -- 172.21.15.114:0/2777572822 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7ff13c051fd0 con 0x7ff14c06f8d0 2024-03-29T10:44:11.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.609+0000 7ff1337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:11.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.609+0000 7ff1337fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:11.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.612+0000 7ff1516a5640 10 librados: watch_flush enter 2024-03-29T10:44:11.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.615+0000 7ff1516a5640 10 librados: watch_flush exit 2024-03-29T10:44:11.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.615+0000 7ff1516a5640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T10:44:11.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.615+0000 7ff1516a5640 10 client.4445.objecter successfully canceled tick 2024-03-29T10:44:11.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.615+0000 7ff1516a5640 1 -- 172.21.15.114:0/2777572822 >> v1:172.21.15.114:6816/775044307 conn(0x7ff13806df70 legacy=0x7ff138070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:11.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.615+0000 7ff1516a5640 10 monclient: shutdown 2024-03-29T10:44:11.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.615+0000 7ff1516a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:11.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.615+0000 7ff1516a5640 1 -- 172.21.15.114:0/2777572822 >> v1:172.21.15.114:6790/0 conn(0x7ff14c06f8d0 legacy=0x7ff14c1eedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:11.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.615+0000 7ff1516a5640 1 -- 172.21.15.114:0/2777572822 shutdown_connections 2024-03-29T10:44:11.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.615+0000 7ff1516a5640 1 -- 172.21.15.114:0/2777572822 >> 172.21.15.114:0/2777572822 conn(0x7ff14c066a00 msgr2=0x7ff14c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:11.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.615+0000 7ff1516a5640 1 -- 172.21.15.114:0/2777572822 shutdown_connections 2024-03-29T10:44:11.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.615+0000 7ff1516a5640 1 -- 172.21.15.114:0/2777572822 wait complete. 2024-03-29T10:44:11.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:11.616+0000 7ff1516a5640 1 librados: shutdown 2024-03-29T10:44:11.617 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:18.424 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:18.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.590+0000 7fd5fb6f4640 10 monclient: get_monmap_and_config 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.590+0000 7fd5fb6f4640 10 monclient: build_initial_monmap 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.590+0000 7fd5fb6f4640 10 monclient: monmap: 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:44:18.591809+0000 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:44:18.591809+0000 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:44:18.575 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.592+0000 7fd5fb6f4640 1 Processor -- start 2024-03-29T10:44:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.592+0000 7fd5fb6f4640 1 -- start start 2024-03-29T10:44:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.592+0000 7fd5fb6f4640 10 monclient: init 2024-03-29T10:44:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.593+0000 7fd5fb6f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.593+0000 7fd5fb6f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.593+0000 7fd5fb6f4640 10 monclient(hunting): picked mon.noname-c con 0x7fd5f40a9ac0 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.593+0000 7fd5fb6f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd5f413ee90 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.593+0000 7fd5fb6f4640 10 monclient(hunting): picked mon.noname-a con 0x7fd5f40af900 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5fb6f4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f40afd10 con 0x7fd5f40af900 2024-03-29T10:44:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5fb6f4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f40af6f0 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5fb6f4640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f40aac80 con 0x7fd5f413ee90 2024-03-29T10:44:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5fb6f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5fb6f4640 10 monclient(hunting): authenticate will time out at 1123.838250s 2024-03-29T10:44:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5f8c8f640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fd5f413ee90 0x7fd5f413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34853/0 (socket says 172.21.15.114:34853) 2024-03-29T10:44:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5f8c8f640 1 -- 172.21.15.114:0/3866528619 learned_addr learned my addr 172.21.15.114:0/3866528619 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 1 -- 172.21.15.114:0/3866528619 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721215102 0 0) 0x7fd5f40aac80 con 0x7fd5f413ee90 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 1 -- 172.21.15.114:0/3866528619 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e0003520 con 0x7fd5f413ee90 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 1 -- 172.21.15.114:0/3866528619 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514121222 0 0) 0x7fd5f40afd10 con 0x7fd5f40af900 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 1 -- 172.21.15.114:0/3866528619 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5f40aac80 con 0x7fd5f40af900 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.597+0000 7fd5ebfff640 1 -- 172.21.15.114:0/3866528619 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3458914307 0 0) 0x7fd5e0003520 con 0x7fd5f413ee90 2024-03-29T10:44:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 1 -- 172.21.15.114:0/3866528619 >> v1:172.21.15.176:6789/0 conn(0x7fd5f40a9ac0 legacy=0x7fd5f40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 1 -- 172.21.15.114:0/3866528619 >> v1:172.21.15.114:6789/0 conn(0x7fd5f40af900 legacy=0x7fd5f4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 1 monclient: found mon.noname-b 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 1 -- 172.21.15.114:0/3866528619 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f40aaf60 con 0x7fd5f413ee90 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 10 monclient: _finish_auth 0 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 1 -- 172.21.15.114:0/3866528619 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd5ec001690 con 0x7fd5f413ee90 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5ebfff640 10 monclient: dump: 2024-03-29T10:44:18.583 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:18.584 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:18.584 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:18.584 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:18.584 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:18.584 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:18.584 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:18.584 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:18.584 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:18.584 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:18.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5fb6f4640 5 monclient: authenticate success, global_id 4448 2024-03-29T10:44:18.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.598+0000 7fd5fb6f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.601+0000 7fd5ebfff640 1 -- 172.21.15.114:0/3866528619 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5ec0017f0 con 0x7fd5f413ee90 2024-03-29T10:44:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.601+0000 7fd5ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.601+0000 7fd5fb6f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.601+0000 7fd5fb6f4640 4 set_mon_vals no callback set 2024-03-29T10:44:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.601+0000 7fd5ebfff640 1 -- 172.21.15.114:0/3866528619 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd5ec001c00 con 0x7fd5f413ee90 2024-03-29T10:44:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.601+0000 7fd5ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.601+0000 7fd5ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.601+0000 7fd5ebfff640 10 monclient: dump: 2024-03-29T10:44:18.585 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:18.585 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:18.586 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:18.586 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:18.586 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:18.586 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:18.586 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:18.586 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:18.586 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:18.586 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.602+0000 7fd5fb6f4640 10 monclient: shutdown 2024-03-29T10:44:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.602+0000 7fd5fb6f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.602+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/3866528619 >> v1:172.21.15.114:6790/0 conn(0x7fd5f413ee90 legacy=0x7fd5f413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.602+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/3866528619 shutdown_connections 2024-03-29T10:44:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.602+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/3866528619 >> 172.21.15.114:0/3866528619 conn(0x7fd5f40b35e0 msgr2=0x7fd5f40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.603+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/3866528619 shutdown_connections 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.603+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/3866528619 wait complete. 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.604+0000 7fd5fb6f4640 10 monclient: build_initial_monmap 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.604+0000 7fd5fb6f4640 10 monclient: monmap: 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:18.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.604+0000 7fd5fb6f4640 1 librados: starting msgr at 2024-03-29T10:44:18.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.604+0000 7fd5fb6f4640 1 librados: starting objecter 2024-03-29T10:44:18.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.604+0000 7fd5fb6f4640 1 Processor -- start 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.605+0000 7fd5fb6f4640 1 -- start start 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.605+0000 7fd5fb6f4640 1 librados: setting wanted keys 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.605+0000 7fd5fb6f4640 1 librados: calling monclient init 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.605+0000 7fd5fb6f4640 10 monclient: init 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.606+0000 7fd5fb6f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.606+0000 7fd5fb6f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.606+0000 7fd5fb6f4640 10 monclient(hunting): picked mon.noname-c con 0x7fd5f40a9ac0 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.606+0000 7fd5fb6f4640 10 monclient(hunting): picked mon.noname-a con 0x7fd5f40af900 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.606+0000 7fd5fb6f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd5f413ee90 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.606+0000 7fd5fb6f4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f41c83c0 con 0x7fd5f40af900 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.606+0000 7fd5fb6f4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f41c9540 con 0x7fd5f413ee90 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.606+0000 7fd5fb6f4640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f41c7790 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.606+0000 7fd5fb6f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:18.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.606+0000 7fd5fb6f4640 10 monclient(hunting): authenticate will time out at 1123.847443s 2024-03-29T10:44:18.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5f9490640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fd5f40a9ac0 0x7fd5f41c7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34911/0 (socket says 172.21.15.114:34911) 2024-03-29T10:44:18.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5f9490640 1 -- 172.21.15.114:0/815525501 learned_addr learned my addr 172.21.15.114:0/815525501 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:18.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 10 client.?.objecter ms_handle_connect 0x7fd5f40a9ac0 2024-03-29T10:44:18.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:18.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877623616 0 0) 0x7fd5f41c7790 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:18.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:18.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:18.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:44:18.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d8004eb0 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2826767326 0 0) 0x7fd5d8004eb0 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5f41c7790 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd5f0001db0 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.612+0000 7fd5e9ffb640 10 monclient(hunting): dump: 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:18.599 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.613+0000 7fd5e9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.614+0000 7fd5e9ffb640 10 client.?.objecter ms_handle_connect 0x7fd5f413ee90 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.614+0000 7fd5e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.614+0000 7fd5e9ffb640 10 client.?.objecter ms_handle_connect 0x7fd5f40af900 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.614+0000 7fd5e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.614+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1973761111 0 0) 0x7fd5f41c7790 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.615+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 >> v1:172.21.15.176:6789/0 conn(0x7fd5f413ee90 legacy=0x7fd5f41dda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.615+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 >> v1:172.21.15.114:6789/0 conn(0x7fd5f40af900 legacy=0x7fd5f41d2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.615+0000 7fd5e9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.615+0000 7fd5e9ffb640 1 monclient: found mon.c 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.615+0000 7fd5e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:18.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.615+0000 7fd5e9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:18.601 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.615+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f40724b0 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.601 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.615+0000 7fd5e9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:18.601 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.615+0000 7fd5e9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:18.601 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.615+0000 7fd5e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:18.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.616+0000 7fd5fb6f4640 5 monclient: authenticate success, global_id 4451 2024-03-29T10:44:18.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.616+0000 7fd5fb6f4640 10 monclient: _renew_subs 2024-03-29T10:44:18.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.616+0000 7fd5fb6f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:18.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.616+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/815525501 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f4071fb0 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.617+0000 7fd5fb6f4640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:18.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.617+0000 7fd5fb6f4640 10 monclient: _renew_subs 2024-03-29T10:44:18.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.617+0000 7fd5fb6f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:18.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.617+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/815525501 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f41d37e0 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.617+0000 7fd5fb6f4640 1 librados: init done 2024-03-29T10:44:18.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.620+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5f00020b0 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.620+0000 7fd5e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:18.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.620+0000 7fd5eb7fe640 4 set_mon_vals no callback set 2024-03-29T10:44:18.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.622+0000 7fd5fb6f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:18.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.622+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fd5f0003de0 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.622+0000 7fd5e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:18.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.622+0000 7fd5e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:18.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.622+0000 7fd5e9ffb640 10 monclient: dump: 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.622+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fd5f0003fc0 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.622+0000 7fd5fb6f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.622+0000 7fd5fb6f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:18.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.622+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/815525501 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d4005080 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.622+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fd5f0085020 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.622+0000 7fd5e9ffb640 10 client.4451.objecter ms_dispatch 0x7fd5f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:18.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.623+0000 7fd5e9ffb640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:18.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.623+0000 7fd5e9ffb640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:18.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.623+0000 7fd5f8c8f640 10 monclient: get_auth_request con 0x7fd5d806db40 auth_method 0 2024-03-29T10:44:18.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.623+0000 7fd5e9ffb640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T10:44:18.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.625+0000 7fd5e9ffb640 10 client.4451.objecter ms_handle_connect 0x7fd5d806db40 2024-03-29T10:44:18.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.632+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5f0050980 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.616 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.632+0000 7fd5e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:18.616 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.632+0000 7fd5e9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:18.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.954+0000 7fd5fb6f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:18.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.954+0000 7fd5fb6f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:18.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.954+0000 7fd5fb6f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:18.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.954+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/815525501 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5d4005330 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.955+0000 7fd5e9ffb640 1 -- 172.21.15.114:0/815525501 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fd5f00528b0 con 0x7fd5f40a9ac0 2024-03-29T10:44:18.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.955+0000 7fd5e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:18.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.955+0000 7fd5e9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:18.939 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:44:18.939 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:18.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.959+0000 7fd5fb6f4640 10 librados: watch_flush enter 2024-03-29T10:44:18.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.959+0000 7fd5fb6f4640 10 librados: watch_flush exit 2024-03-29T10:44:18.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.959+0000 7fd5fb6f4640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T10:44:18.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.959+0000 7fd5fb6f4640 10 client.4451.objecter successfully canceled tick 2024-03-29T10:44:18.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.960+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/815525501 >> v1:172.21.15.114:6816/775044307 conn(0x7fd5d806db40 legacy=0x7fd5d8070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.960+0000 7fd5fb6f4640 10 monclient: shutdown 2024-03-29T10:44:18.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.960+0000 7fd5fb6f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:18.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.960+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/815525501 >> v1:172.21.15.114:6790/0 conn(0x7fd5f40a9ac0 legacy=0x7fd5f41c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.960+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/815525501 shutdown_connections 2024-03-29T10:44:18.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.960+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/815525501 >> 172.21.15.114:0/815525501 conn(0x7fd5f40b35e0 msgr2=0x7fd5f40ab760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:18.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.960+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/815525501 shutdown_connections 2024-03-29T10:44:18.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.960+0000 7fd5fb6f4640 1 -- 172.21.15.114:0/815525501 wait complete. 2024-03-29T10:44:18.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:18.960+0000 7fd5fb6f4640 1 librados: shutdown 2024-03-29T10:44:18.960 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:25.767 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:25.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.939+0000 7f1c5329a640 10 monclient: get_monmap_and_config 2024-03-29T10:44:25.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.939+0000 7f1c5329a640 10 monclient: build_initial_monmap 2024-03-29T10:44:25.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.939+0000 7f1c5329a640 10 monclient: monmap: 2024-03-29T10:44:25.922 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:25.922 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:25.922 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:44:25.940218+0000 2024-03-29T10:44:25.922 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:44:25.940218+0000 2024-03-29T10:44:25.922 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:25.922 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:25.923 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:25.923 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:44:25.923 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:44:25.923 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:25.925 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.941+0000 7f1c5329a640 1 Processor -- start 2024-03-29T10:44:25.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.941+0000 7f1c5329a640 1 -- start start 2024-03-29T10:44:25.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.941+0000 7f1c5329a640 10 monclient: init 2024-03-29T10:44:25.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.943+0000 7f1c5329a640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:25.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.943+0000 7f1c5329a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:25.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.946+0000 7f1c5329a640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4c0b4330 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:25.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.946+0000 7f1c5329a640 10 monclient(hunting): picked mon.noname-a con 0x7f1c4c1470b0 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:25.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.946+0000 7f1c5329a640 10 monclient(hunting): picked mon.noname-b con 0x7f1c4c1439b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:25.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.946+0000 7f1c5329a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c06f2a0 con 0x7f1c4c1470b0 2024-03-29T10:44:25.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.946+0000 7f1c5329a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c146b90 con 0x7f1c4c0b4330 2024-03-29T10:44:25.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.946+0000 7f1c5329a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c0b2ca0 con 0x7f1c4c1439b0 2024-03-29T10:44:25.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.946+0000 7f1c5329a640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:25.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.946+0000 7f1c5329a640 10 monclient(hunting): authenticate will time out at 1131.186616s 2024-03-29T10:44:25.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.947+0000 7f1c51036640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f1c4c0b4330 0x7f1c4c0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:53781/0 (socket says 172.21.15.114:53781) 2024-03-29T10:44:25.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.947+0000 7f1c51036640 1 -- 172.21.15.114:0/2542687201 learned_addr learned my addr 172.21.15.114:0/2542687201 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:25.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037131067 0 0) 0x7f1c4c0b2ca0 con 0x7f1c4c1439b0 2024-03-29T10:44:25.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:25.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:25.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:25.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c38003520 con 0x7f1c4c1439b0 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711260428 0 0) 0x7f1c4c06f2a0 con 0x7f1c4c1470b0 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c4c0b2ca0 con 0x7f1c4c1470b0 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209752538 0 0) 0x7f1c4c146b90 con 0x7f1c4c0b4330 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:44:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.958+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c4c06f2a0 con 0x7f1c4c0b4330 2024-03-29T10:44:25.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3298361760 0 0) 0x7f1c4c0b2ca0 con 0x7f1c4c1470b0 2024-03-29T10:44:25.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 >> v1:172.21.15.114:6790/0 conn(0x7f1c4c1439b0 legacy=0x7f1c4c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:25.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 >> v1:172.21.15.176:6789/0 conn(0x7f1c4c0b4330 legacy=0x7f1c4c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:25.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 1 monclient: found mon.noname-a 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c06e0c0 con 0x7f1c4c1470b0 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1c3c0015f0 con 0x7f1c4c1470b0 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c43fff640 10 monclient: dump: 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:25.945 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:25.946 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:25.946 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:25.946 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:25.946 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:25.946 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:25.946 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:25.946 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:25.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c5329a640 5 monclient: authenticate success, global_id 4419 2024-03-29T10:44:25.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.962+0000 7f1c5329a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:25.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.963+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c3c001750 con 0x7f1c4c1470b0 2024-03-29T10:44:25.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.963+0000 7f1c43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:25.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.963+0000 7f1c5329a640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:25.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.963+0000 7f1c5329a640 4 set_mon_vals no callback set 2024-03-29T10:44:25.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.963+0000 7f1c5329a640 10 monclient: shutdown 2024-03-29T10:44:25.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.963+0000 7f1c5329a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:25.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.963+0000 7f1c5329a640 1 -- 172.21.15.114:0/2542687201 >> v1:172.21.15.114:6789/0 conn(0x7f1c4c1470b0 legacy=0x7f1c4c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:25.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.963+0000 7f1c43fff640 1 -- 172.21.15.114:0/2542687201 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1c3c001ba0 con 0x7f1c4c1470b0 2024-03-29T10:44:25.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.964+0000 7f1c43fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:44:25.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.964+0000 7f1c5329a640 1 -- 172.21.15.114:0/2542687201 shutdown_connections 2024-03-29T10:44:25.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.964+0000 7f1c5329a640 1 -- 172.21.15.114:0/2542687201 >> 172.21.15.114:0/2542687201 conn(0x7f1c4c066a00 msgr2=0x7f1c4c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:25.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.964+0000 7f1c5329a640 1 -- 172.21.15.114:0/2542687201 shutdown_connections 2024-03-29T10:44:25.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.964+0000 7f1c5329a640 1 -- 172.21.15.114:0/2542687201 wait complete. 2024-03-29T10:44:25.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.965+0000 7f1c5329a640 10 monclient: build_initial_monmap 2024-03-29T10:44:25.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.965+0000 7f1c5329a640 10 monclient: monmap: 2024-03-29T10:44:25.951 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:25.951 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:25.951 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:44:25.951 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.965+0000 7f1c5329a640 1 librados: starting msgr at 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.965+0000 7f1c5329a640 1 librados: starting objecter 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.965+0000 7f1c5329a640 1 Processor -- start 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.966+0000 7f1c5329a640 1 -- start start 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.966+0000 7f1c5329a640 1 librados: setting wanted keys 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.966+0000 7f1c5329a640 1 librados: calling monclient init 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.966+0000 7f1c5329a640 10 monclient: init 2024-03-29T10:44:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.966+0000 7f1c5329a640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.966+0000 7f1c5329a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.969+0000 7f1c5329a640 10 monclient(hunting): picked mon.noname-b con 0x7f1c4c0b4330 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.969+0000 7f1c5329a640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4c1439b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.969+0000 7f1c5329a640 10 monclient(hunting): picked mon.noname-a con 0x7f1c4c1470b0 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.969+0000 7f1c5329a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c1ec390 con 0x7f1c4c1470b0 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.969+0000 7f1c5329a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c1f0270 con 0x7f1c4c0b4330 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.969+0000 7f1c5329a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c1ecdf0 con 0x7f1c4c1439b0 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.969+0000 7f1c5329a640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.969+0000 7f1c5329a640 10 monclient(hunting): authenticate will time out at 1131.209800s 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.969+0000 7f1c51036640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f1c4c0b4330 0x7f1c4c1ec6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:53891/0 (socket says 172.21.15.114:53891) 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.969+0000 7f1c51036640 1 -- 172.21.15.114:0/3046661650 learned_addr learned my addr 172.21.15.114:0/3046661650 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.970+0000 7f1c41ffb640 10 client.?.objecter ms_handle_connect 0x7f1c4c0b4330 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.970+0000 7f1c41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.970+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508386590 0 0) 0x7f1c4c1f0270 con 0x7f1c4c0b4330 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.970+0000 7f1c41ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.970+0000 7f1c41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.970+0000 7f1c41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.970+0000 7f1c41ffb640 10 monclient(hunting): my global_id is 4399 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.970+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c30004eb0 con 0x7f1c4c0b4330 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.971+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3104229904 0 0) 0x7f1c30004eb0 con 0x7f1c4c0b4330 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.971+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c4c1f0270 con 0x7f1c4c0b4330 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.971+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1c48001870 con 0x7f1c4c0b4330 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.971+0000 7f1c41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.971+0000 7f1c41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.971+0000 7f1c41ffb640 10 monclient(hunting): dump: 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:25.955 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:25.956 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:25.956 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:25.956 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:25.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.971+0000 7f1c41ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:25.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.971+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 753335863 0 0) 0x7f1c4c1f0270 con 0x7f1c4c0b4330 2024-03-29T10:44:25.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.971+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 >> v1:172.21.15.114:6790/0 conn(0x7f1c4c1439b0 legacy=0x7f1c4c1eb570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:25.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 >> v1:172.21.15.114:6789/0 conn(0x7f1c4c1470b0 legacy=0x7f1c4c1ebc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 1 monclient: found mon.b 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c1f0790 con 0x7f1c4c0b4330 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c5329a640 5 monclient: authenticate success, global_id 4399 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c5329a640 10 monclient: _renew_subs 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c5329a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c5329a640 1 -- 172.21.15.114:0/3046661650 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4c1f09e0 con 0x7f1c4c0b4330 2024-03-29T10:44:25.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c480019d0 con 0x7f1c4c0b4330 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1c48004a40 con 0x7f1c4c0b4330 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.973+0000 7f1c41ffb640 10 monclient: dump: 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:25.961 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.975+0000 7f1c5329a640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.975+0000 7f1c5329a640 10 monclient: _renew_subs 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.975+0000 7f1c5329a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.975+0000 7f1c5329a640 1 -- 172.21.15.114:0/3046661650 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4c08b5c0 con 0x7f1c4c0b4330 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.975+0000 7f1c5329a640 1 librados: init done 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.975+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f1c48004c60 con 0x7f1c4c0b4330 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.976+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f1c48054a60 con 0x7f1c4c0b4330 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.976+0000 7f1c41ffb640 10 client.4399.objecter ms_dispatch 0x7f1c4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.976+0000 7f1c41ffb640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.976+0000 7f1c41ffb640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.976+0000 7f1c41ffb640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.975+0000 7f1c437fe640 4 set_mon_vals no callback set 2024-03-29T10:44:25.962 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.977+0000 7f1c50835640 10 monclient: get_auth_request con 0x7f1c3006da30 auth_method 0 2024-03-29T10:44:25.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.978+0000 7f1c41ffb640 10 client.4399.objecter ms_handle_connect 0x7f1c3006da30 2024-03-29T10:44:25.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.979+0000 7f1c5329a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:25.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.979+0000 7f1c5329a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:25.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.979+0000 7f1c5329a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:25.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.979+0000 7f1c5329a640 1 -- 172.21.15.114:0/3046661650 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c4c075f10 con 0x7f1c4c0b4330 2024-03-29T10:44:25.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.989+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c480503a0 con 0x7f1c4c0b4330 2024-03-29T10:44:25.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.989+0000 7f1c41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:25.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:25.989+0000 7f1c41ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:26.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.319+0000 7f1c5329a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:26.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.319+0000 7f1c5329a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:26.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.319+0000 7f1c5329a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:26.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.319+0000 7f1c5329a640 1 -- 172.21.15.114:0/3046661650 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c4c1f0400 con 0x7f1c4c0b4330 2024-03-29T10:44:26.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.321+0000 7f1c41ffb640 1 -- 172.21.15.114:0/3046661650 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f1c480524a0 con 0x7f1c4c0b4330 2024-03-29T10:44:26.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.321+0000 7f1c41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:26.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.321+0000 7f1c41ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:26.306 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:44:26.306 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:26.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.324+0000 7f1c5329a640 10 librados: watch_flush enter 2024-03-29T10:44:26.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.324+0000 7f1c5329a640 10 librados: watch_flush exit 2024-03-29T10:44:26.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.324+0000 7f1c5329a640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-29T10:44:26.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.324+0000 7f1c5329a640 10 client.4399.objecter successfully canceled tick 2024-03-29T10:44:26.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.325+0000 7f1c5329a640 1 -- 172.21.15.114:0/3046661650 >> v1:172.21.15.114:6816/775044307 conn(0x7f1c3006da30 legacy=0x7f1c3006fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:26.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.325+0000 7f1c5329a640 10 monclient: shutdown 2024-03-29T10:44:26.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.325+0000 7f1c5329a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:26.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.325+0000 7f1c5329a640 1 -- 172.21.15.114:0/3046661650 >> v1:172.21.15.176:6789/0 conn(0x7f1c4c0b4330 legacy=0x7f1c4c1ec6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:26.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.325+0000 7f1c5329a640 1 -- 172.21.15.114:0/3046661650 shutdown_connections 2024-03-29T10:44:26.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.325+0000 7f1c5329a640 1 -- 172.21.15.114:0/3046661650 >> 172.21.15.114:0/3046661650 conn(0x7f1c4c066a00 msgr2=0x7f1c4c0b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:26.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.326+0000 7f1c5329a640 1 -- 172.21.15.114:0/3046661650 shutdown_connections 2024-03-29T10:44:26.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.326+0000 7f1c5329a640 1 -- 172.21.15.114:0/3046661650 wait complete. 2024-03-29T10:44:26.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:26.326+0000 7f1c5329a640 1 librados: shutdown 2024-03-29T10:44:26.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:33.138 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:33.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.328+0000 7f7f5b59e640 10 monclient: get_monmap_and_config 2024-03-29T10:44:33.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.328+0000 7f7f5b59e640 10 monclient: build_initial_monmap 2024-03-29T10:44:33.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.328+0000 7f7f5b59e640 10 monclient: monmap: 2024-03-29T10:44:33.312 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:33.312 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:33.312 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:44:33.329685+0000 2024-03-29T10:44:33.312 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:44:33.329685+0000 2024-03-29T10:44:33.312 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:33.312 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:33.312 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:33.312 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:44:33.312 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:44:33.312 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.329+0000 7f7f5b59e640 1 Processor -- start 2024-03-29T10:44:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.329+0000 7f7f5b59e640 1 -- start start 2024-03-29T10:44:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.329+0000 7f7f5b59e640 10 monclient: init 2024-03-29T10:44:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.330+0000 7f7f5b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.330+0000 7f7f5b59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.330+0000 7f7f5b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f7f54036a60 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.330+0000 7f7f5b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f7f540c5de0 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.330+0000 7f7f5b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f7f540c3070 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.330+0000 7f7f5b59e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5402cc60 con 0x7f7f54036a60 2024-03-29T10:44:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.330+0000 7f7f5b59e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5402e040 con 0x7f7f540c5de0 2024-03-29T10:44:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.330+0000 7f7f5b59e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5402e320 con 0x7f7f540c3070 2024-03-29T10:44:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.330+0000 7f7f5b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.330+0000 7f7f5b59e640 10 monclient(hunting): authenticate will time out at 1138.571837s 2024-03-29T10:44:33.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.334+0000 7f7f59d9b640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f7f540c5de0 0x7f7f540c2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:32905/0 (socket says 172.21.15.114:32905) 2024-03-29T10:44:33.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.334+0000 7f7f59d9b640 1 -- 172.21.15.114:0/3832055762 learned_addr learned my addr 172.21.15.114:0/3832055762 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:33.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 1 -- 172.21.15.114:0/3832055762 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178313405 0 0) 0x7f7f5402e040 con 0x7f7f540c5de0 2024-03-29T10:44:33.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:33.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:33.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:33.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:44:33.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 1 -- 172.21.15.114:0/3832055762 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f44003520 con 0x7f7f540c5de0 2024-03-29T10:44:33.322 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 1 -- 172.21.15.114:0/3832055762 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2425982143 0 0) 0x7f7f44003520 con 0x7f7f540c5de0 2024-03-29T10:44:33.322 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 1 -- 172.21.15.114:0/3832055762 >> v1:172.21.15.114:6790/0 conn(0x7f7f540c3070 legacy=0x7f7f540d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:33.322 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 1 -- 172.21.15.114:0/3832055762 >> v1:172.21.15.114:6789/0 conn(0x7f7f54036a60 legacy=0x7f7f540c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:33.322 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 10 monclient: _finish_hunting 0 2024-03-29T10:44:33.322 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 1 monclient: found mon.noname-c 2024-03-29T10:44:33.322 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:33.322 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 1 -- 172.21.15.114:0/3832055762 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5402c830 con 0x7f7f540c5de0 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 10 monclient: _finish_auth 0 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 10 monclient: _check_auth_tickets 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5b59e640 5 monclient: authenticate success, global_id 4402 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 1 -- 172.21.15.114:0/3832055762 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7f5c04f730 con 0x7f7f540c5de0 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5959a640 10 monclient: dump: 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:33.323 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:33.324 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:33.324 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:33.324 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:33.324 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:33.324 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:33.324 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:33.324 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:33.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.338+0000 7f7f5b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:33.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.339+0000 7f7f5959a640 1 -- 172.21.15.114:0/3832055762 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f5c0008d0 con 0x7f7f540c5de0 2024-03-29T10:44:33.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.339+0000 7f7f5959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:33.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.339+0000 7f7f5b59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:33.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.339+0000 7f7f5b59e640 4 set_mon_vals no callback set 2024-03-29T10:44:33.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.340+0000 7f7f5b59e640 10 monclient: shutdown 2024-03-29T10:44:33.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.340+0000 7f7f5b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:33.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.340+0000 7f7f5b59e640 1 -- 172.21.15.114:0/3832055762 >> v1:172.21.15.176:6789/0 conn(0x7f7f540c5de0 legacy=0x7f7f540c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:33.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.340+0000 7f7f5b59e640 1 -- 172.21.15.114:0/3832055762 shutdown_connections 2024-03-29T10:44:33.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.340+0000 7f7f5b59e640 1 -- 172.21.15.114:0/3832055762 >> 172.21.15.114:0/3832055762 conn(0x7f7f54005c00 msgr2=0x7f7f54006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:33.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.343+0000 7f7f5b59e640 1 -- 172.21.15.114:0/3832055762 shutdown_connections 2024-03-29T10:44:33.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.343+0000 7f7f5b59e640 1 -- 172.21.15.114:0/3832055762 wait complete. 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.344+0000 7f7f5b59e640 10 monclient: build_initial_monmap 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.344+0000 7f7f5b59e640 10 monclient: monmap: 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:44:33.328 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:33.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.345+0000 7f7f5b59e640 1 librados: starting msgr at 2024-03-29T10:44:33.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.345+0000 7f7f5b59e640 1 librados: starting objecter 2024-03-29T10:44:33.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.345+0000 7f7f5b59e640 1 Processor -- start 2024-03-29T10:44:33.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.345+0000 7f7f5b59e640 1 -- start start 2024-03-29T10:44:33.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.345+0000 7f7f5b59e640 1 librados: setting wanted keys 2024-03-29T10:44:33.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.345+0000 7f7f5b59e640 1 librados: calling monclient init 2024-03-29T10:44:33.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.345+0000 7f7f5b59e640 10 monclient: init 2024-03-29T10:44:33.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.346+0000 7f7f5b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.346+0000 7f7f5b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.346+0000 7f7f5b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f7f54036a60 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.346+0000 7f7f5b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f7f540c3070 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.346+0000 7f7f5b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f7f540bc800 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.346+0000 7f7f5b59e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f540c8fd0 con 0x7f7f540c3070 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.346+0000 7f7f5b59e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f540c8350 con 0x7f7f54036a60 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.346+0000 7f7f5b59e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f540c84e0 con 0x7f7f540bc800 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.346+0000 7f7f5b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.346+0000 7f7f5b59e640 10 monclient(hunting): authenticate will time out at 1138.587211s 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.350+0000 7f7f59d9b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7f540c3070 0x7f7f541582c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52164/0 (socket says 172.21.15.114:52164) 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.350+0000 7f7f59d9b640 1 -- 172.21.15.114:0/4245952186 learned_addr learned my addr 172.21.15.114:0/4245952186 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.351+0000 7f7f4affd640 10 client.?.objecter ms_handle_connect 0x7f7f540c3070 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.351+0000 7f7f4affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:33.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.351+0000 7f7f4affd640 10 client.?.objecter ms_handle_connect 0x7f7f540bc800 2024-03-29T10:44:33.336 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.351+0000 7f7f4affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:33.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 10 client.?.objecter ms_handle_connect 0x7f7f54036a60 2024-03-29T10:44:33.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:33.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309353956 0 0) 0x7f7f540c8350 con 0x7f7f54036a60 2024-03-29T10:44:33.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:33.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:33.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f3c004eb0 con 0x7f7f54036a60 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579699856 0 0) 0x7f7f540c84e0 con 0x7f7f540bc800 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.354+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f540c8350 con 0x7f7f540bc800 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3571869045 0 0) 0x7f7f3c004eb0 con 0x7f7f54036a60 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f540c84e0 con 0x7f7f54036a60 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7f4c001860 con 0x7f7f54036a60 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:33.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient(hunting): dump: 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 926637515 0 0) 0x7f7f540c8350 con 0x7f7f540bc800 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f3c004eb0 con 0x7f7f540bc800 2024-03-29T10:44:33.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7f500018c0 con 0x7f7f540bc800 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient(hunting): dump: 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:33.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3099672326 0 0) 0x7f7f540c84e0 con 0x7f7f54036a60 2024-03-29T10:44:33.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 >> v1:172.21.15.114:6790/0 conn(0x7f7f540bc800 legacy=0x7f7f541589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:33.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 >> v1:172.21.15.114:6789/0 conn(0x7f7f540c3070 legacy=0x7f7f541582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:33.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:33.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 monclient: found mon.b 2024-03-29T10:44:33.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:33.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:33.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f540c8670 con 0x7f7f54036a60 2024-03-29T10:44:33.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient: _finish_auth 0 2024-03-29T10:44:33.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:33.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:33.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f4c001d50 con 0x7f7f54036a60 2024-03-29T10:44:33.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f4affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:33.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.355+0000 7f7f58d99640 4 set_mon_vals no callback set 2024-03-29T10:44:33.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.357+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7f4c0037b0 con 0x7f7f54036a60 2024-03-29T10:44:33.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.357+0000 7f7f4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:33.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.357+0000 7f7f4affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:33.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.357+0000 7f7f4affd640 10 monclient: dump: 2024-03-29T10:44:33.343 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:33.343 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:33.343 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.359+0000 7f7f5b59e640 5 monclient: authenticate success, global_id 4405 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.359+0000 7f7f5b59e640 10 monclient: _renew_subs 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.359+0000 7f7f5b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.359+0000 7f7f5b59e640 1 -- 172.21.15.114:0/4245952186 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f540c4870 con 0x7f7f54036a60 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.359+0000 7f7f5b59e640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:33.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.359+0000 7f7f5b59e640 10 monclient: _renew_subs 2024-03-29T10:44:33.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.359+0000 7f7f5b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:33.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.359+0000 7f7f5b59e640 1 -- 172.21.15.114:0/4245952186 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f540c8900 con 0x7f7f54036a60 2024-03-29T10:44:33.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.359+0000 7f7f5b59e640 1 librados: init done 2024-03-29T10:44:33.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.362+0000 7f7f5b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:33.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.362+0000 7f7f5b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:33.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.362+0000 7f7f5b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:33.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.362+0000 7f7f5b59e640 1 -- 172.21.15.114:0/4245952186 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5402f1e0 con 0x7f7f54036a60 2024-03-29T10:44:33.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.363+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f7f4c019780 con 0x7f7f54036a60 2024-03-29T10:44:33.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.364+0000 7f7f59d9b640 10 monclient: get_auth_request con 0x7f7f3c06dd70 auth_method 0 2024-03-29T10:44:33.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.365+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f7f4c083760 con 0x7f7f54036a60 2024-03-29T10:44:33.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.365+0000 7f7f4affd640 10 client.4405.objecter ms_dispatch 0x7f7f5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:33.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.365+0000 7f7f4affd640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:33.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.365+0000 7f7f4affd640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:33.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.365+0000 7f7f4affd640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T10:44:33.349 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.366+0000 7f7f4affd640 10 client.4405.objecter ms_handle_connect 0x7f7f3c06dd70 2024-03-29T10:44:33.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.370+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f4c04ee70 con 0x7f7f54036a60 2024-03-29T10:44:33.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.370+0000 7f7f4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:33.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.370+0000 7f7f4affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:33.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.712+0000 7f7f5b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:33.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.712+0000 7f7f5b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:33.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.712+0000 7f7f5b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:33.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.712+0000 7f7f5b59e640 1 -- 172.21.15.114:0/4245952186 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f540c9160 con 0x7f7f54036a60 2024-03-29T10:44:33.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.713+0000 7f7f4affd640 1 -- 172.21.15.114:0/4245952186 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f7f4c050f70 con 0x7f7f54036a60 2024-03-29T10:44:33.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.713+0000 7f7f4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:33.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.713+0000 7f7f4affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:33.699 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:44:33.699 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:33.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.719+0000 7f7f5b59e640 10 librados: watch_flush enter 2024-03-29T10:44:33.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.719+0000 7f7f5b59e640 10 librados: watch_flush exit 2024-03-29T10:44:33.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.719+0000 7f7f5b59e640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T10:44:33.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.719+0000 7f7f5b59e640 10 client.4405.objecter successfully canceled tick 2024-03-29T10:44:33.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.721+0000 7f7f5b59e640 1 -- 172.21.15.114:0/4245952186 >> v1:172.21.15.114:6816/775044307 conn(0x7f7f3c06dd70 legacy=0x7f7f3c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:33.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.721+0000 7f7f5b59e640 10 monclient: shutdown 2024-03-29T10:44:33.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.721+0000 7f7f5b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:33.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.721+0000 7f7f5b59e640 1 -- 172.21.15.114:0/4245952186 >> v1:172.21.15.176:6789/0 conn(0x7f7f54036a60 legacy=0x7f7f540bc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:33.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.721+0000 7f7f5b59e640 1 -- 172.21.15.114:0/4245952186 shutdown_connections 2024-03-29T10:44:33.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.721+0000 7f7f5b59e640 1 -- 172.21.15.114:0/4245952186 >> 172.21.15.114:0/4245952186 conn(0x7f7f54005c00 msgr2=0x7f7f540bf3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:33.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.721+0000 7f7f5b59e640 1 -- 172.21.15.114:0/4245952186 shutdown_connections 2024-03-29T10:44:33.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.721+0000 7f7f5b59e640 1 -- 172.21.15.114:0/4245952186 wait complete. 2024-03-29T10:44:33.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:33.721+0000 7f7f5b59e640 1 librados: shutdown 2024-03-29T10:44:33.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:40.529 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:40.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.694+0000 7fcb8ea82640 10 monclient: get_monmap_and_config 2024-03-29T10:44:40.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.694+0000 7fcb8ea82640 10 monclient: build_initial_monmap 2024-03-29T10:44:40.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.694+0000 7fcb8ea82640 10 monclient: monmap: 2024-03-29T10:44:40.677 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:40.678 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:40.678 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:44:40.695684+0000 2024-03-29T10:44:40.678 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:44:40.695684+0000 2024-03-29T10:44:40.678 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:40.678 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:40.678 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:40.678 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:44:40.678 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:44:40.678 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:40.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.695+0000 7fcb8ea82640 1 Processor -- start 2024-03-29T10:44:40.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.696+0000 7fcb8ea82640 1 -- start start 2024-03-29T10:44:40.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.696+0000 7fcb8ea82640 10 monclient: init 2024-03-29T10:44:40.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.696+0000 7fcb8ea82640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:40.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.696+0000 7fcb8ea82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:40.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.697+0000 7fcb8ea82640 10 monclient(hunting): picked mon.noname-b con 0x7fcb881457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:40.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.697+0000 7fcb8ea82640 10 monclient(hunting): picked mon.noname-c con 0x7fcb8813f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:40.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.697+0000 7fcb8ea82640 10 monclient(hunting): picked mon.noname-a con 0x7fcb880a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:40.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.697+0000 7fcb8ea82640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb880aaef0 con 0x7fcb880a9d70 2024-03-29T10:44:40.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.697+0000 7fcb8ea82640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb88145bd0 con 0x7fcb8813f650 2024-03-29T10:44:40.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.697+0000 7fcb8ea82640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb88145eb0 con 0x7fcb881457f0 2024-03-29T10:44:40.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.697+0000 7fcb8ea82640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:40.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.697+0000 7fcb8ea82640 10 monclient(hunting): authenticate will time out at 1145.938355s 2024-03-29T10:44:40.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.697+0000 7fcb87fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fcb8813f650 0x7fcb880b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:44430/0 (socket says 172.21.15.114:44430) 2024-03-29T10:44:40.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.697+0000 7fcb87fff640 1 -- 172.21.15.114:0/276044732 learned_addr learned my addr 172.21.15.114:0/276044732 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:40.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 1 -- 172.21.15.114:0/276044732 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110464263 0 0) 0x7fcb88145bd0 con 0x7fcb8813f650 2024-03-29T10:44:40.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:40.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:40.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 1 -- 172.21.15.114:0/276044732 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb74003520 con 0x7fcb8813f650 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 1 -- 172.21.15.114:0/276044732 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3129702442 0 0) 0x7fcb74003520 con 0x7fcb8813f650 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 1 -- 172.21.15.114:0/276044732 >> v1:172.21.15.114:6790/0 conn(0x7fcb881457f0 legacy=0x7fcb8813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 1 -- 172.21.15.114:0/276044732 >> v1:172.21.15.114:6789/0 conn(0x7fcb880a9d70 legacy=0x7fcb8814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb8d01f640 1 -- 172.21.15.114:0/276044732 >> v1:172.21.15.114:6789/0 conn(0x7fcb880a9d70 legacy=0x7fcb8814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb8d01f640 1 -- 172.21.15.114:0/276044732 >> v1:172.21.15.114:6789/0 conn(0x7fcb880a9d70 legacy=0x7fcb8814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb8d01f640 1 --1- 172.21.15.114:0/276044732 >> v1:172.21.15.114:6789/0 conn(0x7fcb880a9d70 0x7fcb8814d8c0 :-1 s=OPENED pgs=541452 cs=1 l=1).handle_message read tag failed 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb8d01f640 1 --1- 172.21.15.114:0/276044732 >> v1:172.21.15.114:6789/0 conn(0x7fcb880a9d70 0x7fcb8814d8c0 :-1 s=OPENED pgs=541452 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 1 monclient: found mon.noname-c 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:44:40.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 1 -- 172.21.15.114:0/276044732 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb88146340 con 0x7fcb8813f650 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.698+0000 7fcb877fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 10 monclient: ms_handle_reset stray mon v1:172.21.15.114:6789/0 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb8ea82640 5 monclient: authenticate success, global_id 4408 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb8ea82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 1 -- 172.21.15.114:0/276044732 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fcb70001690 con 0x7fcb8813f650 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 10 monclient: dump: 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:40.686 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb8ea82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 1 -- 172.21.15.114:0/276044732 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb700018b0 con 0x7fcb8813f650 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 1 -- 172.21.15.114:0/276044732 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fcb70001c40 con 0x7fcb8813f650 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb877fe640 10 monclient: dump: 2024-03-29T10:44:40.687 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb8ea82640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.699+0000 7fcb8ea82640 4 set_mon_vals no callback set 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.701+0000 7fcb8ea82640 10 monclient: shutdown 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.701+0000 7fcb8ea82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.701+0000 7fcb8ea82640 1 -- 172.21.15.114:0/276044732 >> v1:172.21.15.176:6789/0 conn(0x7fcb8813f650 legacy=0x7fcb880b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:40.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.701+0000 7fcb8ea82640 1 -- 172.21.15.114:0/276044732 shutdown_connections 2024-03-29T10:44:40.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.701+0000 7fcb8ea82640 1 -- 172.21.15.114:0/276044732 >> 172.21.15.114:0/276044732 conn(0x7fcb88066a00 msgr2=0x7fcb88066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:40.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.701+0000 7fcb8ea82640 1 -- 172.21.15.114:0/276044732 shutdown_connections 2024-03-29T10:44:40.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.701+0000 7fcb8ea82640 1 -- 172.21.15.114:0/276044732 wait complete. 2024-03-29T10:44:40.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.702+0000 7fcb8ea82640 10 monclient: build_initial_monmap 2024-03-29T10:44:40.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.702+0000 7fcb8ea82640 10 monclient: monmap: 2024-03-29T10:44:40.696 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:40.696 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:40.696 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:44:40.696 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:44:40.696 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:40.696 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.702+0000 7fcb8ea82640 1 librados: starting msgr at 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.703+0000 7fcb8ea82640 1 librados: starting objecter 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.703+0000 7fcb8ea82640 1 Processor -- start 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.703+0000 7fcb8ea82640 1 -- start start 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.703+0000 7fcb8ea82640 1 librados: setting wanted keys 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.703+0000 7fcb8ea82640 1 librados: calling monclient init 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.703+0000 7fcb8ea82640 10 monclient: init 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.704+0000 7fcb8ea82640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.704+0000 7fcb8ea82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:40.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb8ea82640 10 monclient(hunting): picked mon.noname-a con 0x7fcb880a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb8ea82640 10 monclient(hunting): picked mon.noname-c con 0x7fcb8813f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb8ea82640 10 monclient(hunting): picked mon.noname-b con 0x7fcb881457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb8ea82640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb881387b0 con 0x7fcb880a9d70 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb8ea82640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8806fbd0 con 0x7fcb881457f0 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb8ea82640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8806feb0 con 0x7fcb8813f650 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb8ea82640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb8ea82640 10 monclient(hunting): authenticate will time out at 1145.946126s 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb8d01f640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fcb881457f0 0x7fcb881e6bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:46696/0 (socket says 172.21.15.114:46696) 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb8d01f640 1 -- 172.21.15.114:0/3951869015 learned_addr learned my addr 172.21.15.114:0/3951869015 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb857fa640 10 client.?.objecter ms_handle_connect 0x7fcb881457f0 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb857fa640 10 client.?.objecter ms_handle_connect 0x7fcb8813f650 2024-03-29T10:44:40.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb857fa640 10 client.?.objecter ms_handle_connect 0x7fcb880a9d70 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.705+0000 7fcb857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635134034 0 0) 0x7fcb8806fbd0 con 0x7fcb881457f0 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb68004eb0 con 0x7fcb881457f0 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635797683 0 0) 0x7fcb881387b0 con 0x7fcb880a9d70 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): my global_id is 4428 2024-03-29T10:44:40.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb8806fbd0 con 0x7fcb880a9d70 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891100747 0 0) 0x7fcb8806feb0 con 0x7fcb8813f650 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb881387b0 con 0x7fcb8813f650 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2387326621 0 0) 0x7fcb68004eb0 con 0x7fcb881457f0 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb8806feb0 con 0x7fcb881457f0 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fcb80001a30 con 0x7fcb881457f0 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): dump: 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:40.700 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.706+0000 7fcb857fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3762356654 0 0) 0x7fcb8806feb0 con 0x7fcb881457f0 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 >> v1:172.21.15.114:6790/0 conn(0x7fcb8813f650 legacy=0x7fcb881e34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 >> v1:172.21.15.114:6789/0 conn(0x7fcb880a9d70 legacy=0x7fcb88140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb857fa640 10 monclient: _finish_hunting 0 2024-03-29T10:44:40.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb857fa640 1 monclient: found mon.b 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb881384d0 con 0x7fcb881457f0 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb857fa640 10 monclient: _finish_auth 0 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb857fa640 10 monclient: _check_auth_tickets 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb8ea82640 5 monclient: authenticate success, global_id 4411 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb8ea82640 10 monclient: _renew_subs 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb8ea82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb8ea82640 1 -- 172.21.15.114:0/3951869015 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8806efb0 con 0x7fcb881457f0 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.707+0000 7fcb8ea82640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:40.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.708+0000 7fcb8ea82640 10 monclient: _renew_subs 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.708+0000 7fcb8ea82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.708+0000 7fcb8ea82640 1 -- 172.21.15.114:0/3951869015 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb88142e70 con 0x7fcb881457f0 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.708+0000 7fcb8ea82640 1 librados: init done 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.708+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb80001b90 con 0x7fcb881457f0 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.708+0000 7fcb857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.708+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fcb80003a40 con 0x7fcb881457f0 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.708+0000 7fcb857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.708+0000 7fcb857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.708+0000 7fcb857fa640 10 monclient: dump: 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:40.703 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.708+0000 7fcb86ffd640 4 set_mon_vals no callback set 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.711+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fcb80003ba0 con 0x7fcb881457f0 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.712+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fcb80084ae0 con 0x7fcb881457f0 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.712+0000 7fcb857fa640 10 client.4411.objecter ms_dispatch 0x7fcb88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.712+0000 7fcb857fa640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.712+0000 7fcb857fa640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.712+0000 7fcb8ea82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.712+0000 7fcb857fa640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T10:44:40.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.712+0000 7fcb8ea82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:40.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.712+0000 7fcb8ea82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:40.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.712+0000 7fcb8ea82640 1 -- 172.21.15.114:0/3951869015 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c005080 con 0x7fcb881457f0 2024-03-29T10:44:40.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.712+0000 7fcb8c81e640 10 monclient: get_auth_request con 0x7fcb6806dbe0 auth_method 0 2024-03-29T10:44:40.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.713+0000 7fcb857fa640 10 client.4411.objecter ms_handle_connect 0x7fcb6806dbe0 2024-03-29T10:44:40.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.721+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb800503c0 con 0x7fcb881457f0 2024-03-29T10:44:40.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.721+0000 7fcb857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:40.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:40.721+0000 7fcb857fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:41.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.100+0000 7fcb8ea82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:41.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.100+0000 7fcb8ea82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:41.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.100+0000 7fcb8ea82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:44:41.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.100+0000 7fcb8ea82640 1 -- 172.21.15.114:0/3951869015 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb6c005330 con 0x7fcb881457f0 2024-03-29T10:44:41.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.102+0000 7fcb857fa640 1 -- 172.21.15.114:0/3951869015 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fcb800522f0 con 0x7fcb881457f0 2024-03-29T10:44:41.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.102+0000 7fcb857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:41.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.102+0000 7fcb857fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:41.086 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:44:41.086 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:41.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.106+0000 7fcb8ea82640 10 librados: watch_flush enter 2024-03-29T10:44:41.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.106+0000 7fcb8ea82640 10 librados: watch_flush exit 2024-03-29T10:44:41.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.106+0000 7fcb8ea82640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T10:44:41.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.106+0000 7fcb8ea82640 10 client.4411.objecter successfully canceled tick 2024-03-29T10:44:41.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.106+0000 7fcb8ea82640 1 -- 172.21.15.114:0/3951869015 >> v1:172.21.15.114:6816/775044307 conn(0x7fcb6806dbe0 legacy=0x7fcb680700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:41.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.106+0000 7fcb8ea82640 10 monclient: shutdown 2024-03-29T10:44:41.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.106+0000 7fcb8ea82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:41.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.106+0000 7fcb8ea82640 1 -- 172.21.15.114:0/3951869015 >> v1:172.21.15.176:6789/0 conn(0x7fcb881457f0 legacy=0x7fcb881e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:41.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.106+0000 7fcb8ea82640 1 -- 172.21.15.114:0/3951869015 shutdown_connections 2024-03-29T10:44:41.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.106+0000 7fcb8ea82640 1 -- 172.21.15.114:0/3951869015 >> 172.21.15.114:0/3951869015 conn(0x7fcb88066a00 msgr2=0x7fcb8813cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:41.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.107+0000 7fcb8ea82640 1 -- 172.21.15.114:0/3951869015 shutdown_connections 2024-03-29T10:44:41.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.107+0000 7fcb8ea82640 1 -- 172.21.15.114:0/3951869015 wait complete. 2024-03-29T10:44:41.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:41.107+0000 7fcb8ea82640 1 librados: shutdown 2024-03-29T10:44:41.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:47.917 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:48.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.088+0000 7f93b21e6640 10 monclient: get_monmap_and_config 2024-03-29T10:44:48.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.088+0000 7f93b21e6640 10 monclient: build_initial_monmap 2024-03-29T10:44:48.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.088+0000 7f93b21e6640 10 monclient: monmap: 2024-03-29T10:44:48.071 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:48.071 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:48.072 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:44:48.089508+0000 2024-03-29T10:44:48.072 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:44:48.089508+0000 2024-03-29T10:44:48.072 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:48.072 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:48.072 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:48.072 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:44:48.072 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:44:48.072 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:48.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.089+0000 7f93b21e6640 1 Processor -- start 2024-03-29T10:44:48.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.089+0000 7f93b21e6640 1 -- start start 2024-03-29T10:44:48.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.089+0000 7f93b21e6640 10 monclient: init 2024-03-29T10:44:48.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93b21e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:48.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93b21e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:48.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93b21e6640 10 monclient(hunting): picked mon.noname-a con 0x7f93ac1422b0 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:48.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93b21e6640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac142690 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:48.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93b21e6640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac0b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:48.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93b21e6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac142c80 con 0x7f93ac1422b0 2024-03-29T10:44:48.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93b21e6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac0aae40 con 0x7f93ac0b17d0 2024-03-29T10:44:48.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93b21e6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac13fbd0 con 0x7f93ac142690 2024-03-29T10:44:48.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93b21e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:48.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93b21e6640 10 monclient(hunting): authenticate will time out at 1153.331514s 2024-03-29T10:44:48.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93ab7fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f93ac1422b0 0x7f93ac13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36934/0 (socket says 172.21.15.114:36934) 2024-03-29T10:44:48.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.090+0000 7f93ab7fe640 1 -- 172.21.15.114:0/3667051204 learned_addr learned my addr 172.21.15.114:0/3667051204 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 1 -- 172.21.15.114:0/3667051204 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184175606 0 0) 0x7f93ac13fbd0 con 0x7f93ac142690 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 1 -- 172.21.15.114:0/3667051204 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9398003520 con 0x7f93ac142690 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 1 -- 172.21.15.114:0/3667051204 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830354667 0 0) 0x7f93ac142c80 con 0x7f93ac1422b0 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 10 monclient(hunting): my global_id is 4431 2024-03-29T10:44:48.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.091+0000 7f93aa7fc640 1 -- 172.21.15.114:0/3667051204 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93ac13fbd0 con 0x7f93ac1422b0 2024-03-29T10:44:48.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 1 -- 172.21.15.114:0/3667051204 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224865167 0 0) 0x7f9398003520 con 0x7f93ac142690 2024-03-29T10:44:48.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 1 -- 172.21.15.114:0/3667051204 >> v1:172.21.15.176:6789/0 conn(0x7f93ac0b17d0 legacy=0x7f93ac0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:48.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 1 -- 172.21.15.114:0/3667051204 >> v1:172.21.15.114:6789/0 conn(0x7f93ac1422b0 legacy=0x7f93ac13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 1 monclient: found mon.noname-b 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 1 -- 172.21.15.114:0/3667051204 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac151940 con 0x7f93ac142690 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93b21e6640 5 monclient: authenticate success, global_id 4469 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93b21e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 1 -- 172.21.15.114:0/3667051204 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f93a4001690 con 0x7f93ac142690 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:48.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 10 monclient: dump: 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93b21e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:48.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 1 -- 172.21.15.114:0/3667051204 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93a40018b0 con 0x7f93ac142690 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93b21e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93b21e6640 4 set_mon_vals no callback set 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 1 -- 172.21.15.114:0/3667051204 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f93a4001bc0 con 0x7f93ac142690 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.092+0000 7f93aa7fc640 10 monclient: dump: 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:48.079 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:48.080 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:48.080 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:48.080 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:48.080 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.093+0000 7f93b21e6640 10 monclient: shutdown 2024-03-29T10:44:48.080 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.093+0000 7f93b21e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:48.080 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.093+0000 7f93b21e6640 1 -- 172.21.15.114:0/3667051204 >> v1:172.21.15.114:6790/0 conn(0x7f93ac142690 legacy=0x7f93ac0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:48.080 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.093+0000 7f93b21e6640 1 -- 172.21.15.114:0/3667051204 shutdown_connections 2024-03-29T10:44:48.080 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.093+0000 7f93b21e6640 1 -- 172.21.15.114:0/3667051204 >> 172.21.15.114:0/3667051204 conn(0x7f93ac066a00 msgr2=0x7f93ac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.094+0000 7f93b21e6640 1 -- 172.21.15.114:0/3667051204 shutdown_connections 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.094+0000 7f93b21e6640 1 -- 172.21.15.114:0/3667051204 wait complete. 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.094+0000 7f93b21e6640 10 monclient: build_initial_monmap 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.094+0000 7f93b21e6640 10 monclient: monmap: 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:44:48.084 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.095+0000 7f93b21e6640 1 librados: starting msgr at 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.095+0000 7f93b21e6640 1 librados: starting objecter 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.095+0000 7f93b21e6640 1 Processor -- start 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.095+0000 7f93b21e6640 1 -- start start 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.095+0000 7f93b21e6640 1 librados: setting wanted keys 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.095+0000 7f93b21e6640 1 librados: calling monclient init 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.095+0000 7f93b21e6640 10 monclient: init 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.096+0000 7f93b21e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.096+0000 7f93b21e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.096+0000 7f93b21e6640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac0b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.096+0000 7f93b21e6640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac1422b0 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.096+0000 7f93b21e6640 10 monclient(hunting): picked mon.noname-a con 0x7f93ac142690 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:48.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.096+0000 7f93b21e6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac131ef0 con 0x7f93ac142690 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.096+0000 7f93b21e6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac132ae0 con 0x7f93ac1422b0 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.096+0000 7f93b21e6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac12f680 con 0x7f93ac0b17d0 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.096+0000 7f93b21e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.096+0000 7f93b21e6640 10 monclient(hunting): authenticate will time out at 1153.337983s 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93ab7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f93ac0b17d0 0x7f93ac1323d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:44662/0 (socket says 172.21.15.114:44662) 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93ab7fe640 1 -- 172.21.15.114:0/4279707016 learned_addr learned my addr 172.21.15.114:0/4279707016 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 client.?.objecter ms_handle_connect 0x7f93ac0b17d0 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 client.?.objecter ms_handle_connect 0x7f93ac142690 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285487874 0 0) 0x7f93ac12f680 con 0x7f93ac0b17d0 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:48.086 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f938c004eb0 con 0x7f93ac0b17d0 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93aaffd640 1 -- 172.21.15.114:0/4279707016 >> v1:172.21.15.176:6789/0 conn(0x7f93ac1422b0 legacy=0x7f93ac12e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93aaffd640 1 -- 172.21.15.114:0/4279707016 >> v1:172.21.15.176:6789/0 conn(0x7f93ac1422b0 legacy=0x7f93ac12e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93aaffd640 1 --1- 172.21.15.114:0/4279707016 >> v1:172.21.15.176:6789/0 conn(0x7f93ac1422b0 0x7f93ac12e860 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93aaffd640 1 --1- 172.21.15.114:0/4279707016 >> v1:172.21.15.176:6789/0 conn(0x7f93ac1422b0 0x7f93ac12e860 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.176:6789/0 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488758827 0 0) 0x7f93ac131ef0 con 0x7f93ac142690 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:48.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.097+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93ac12f680 con 0x7f93ac142690 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 312977582 0 0) 0x7f938c004eb0 con 0x7f93ac0b17d0 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93ac131ef0 con 0x7f93ac0b17d0 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f939c001d70 con 0x7f93ac0b17d0 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 10 monclient(hunting): dump: 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:48.088 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3557777460 0 0) 0x7f93ac12f680 con 0x7f93ac142690 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f938c004eb0 con 0x7f93ac142690 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9394001510 con 0x7f93ac142690 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 10 monclient(hunting): dump: 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:48.089 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4203733215 0 0) 0x7f93ac131ef0 con 0x7f93ac0b17d0 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 >> v1:172.21.15.176:6789/0 conn(0x7f93ac1422b0 legacy=0x7f93ac12e860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 >> v1:172.21.15.114:6789/0 conn(0x7f93ac142690 legacy=0x7f93ac12ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 1 monclient: found mon.c 2024-03-29T10:44:48.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac06ee10 con 0x7f93ac0b17d0 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93b21e6640 5 monclient: authenticate success, global_id 4472 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.098+0000 7f93b21e6640 10 monclient: _renew_subs 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93b21e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93b21e6640 1 -- 172.21.15.114:0/4279707016 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ac06de90 con 0x7f93ac0b17d0 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93b21e6640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93b21e6640 10 monclient: _renew_subs 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93b21e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93b21e6640 1 -- 172.21.15.114:0/4279707016 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ac131a60 con 0x7f93ac0b17d0 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93b21e6640 1 librados: init done 2024-03-29T10:44:48.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f939c001ef0 con 0x7f93ac0b17d0 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f939c0020e0 con 0x7f93ac0b17d0 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93a3fff640 10 monclient: dump: 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:48.092 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:48.093 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:48.093 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:48.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.099+0000 7f93a9ffb640 4 set_mon_vals no callback set 2024-03-29T10:44:48.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.100+0000 7f93b21e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:48.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.100+0000 7f93b21e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:48.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.100+0000 7f93b21e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:48.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.100+0000 7f93b21e6640 1 -- 172.21.15.114:0/4279707016 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9390005080 con 0x7f93ac0b17d0 2024-03-29T10:44:48.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.102+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f939c01ae20 con 0x7f93ac0b17d0 2024-03-29T10:44:48.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.102+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f939c084d60 con 0x7f93ac0b17d0 2024-03-29T10:44:48.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.102+0000 7f93a3fff640 10 client.4472.objecter ms_dispatch 0x7f93ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:48.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.102+0000 7f93a3fff640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:48.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.102+0000 7f93a3fff640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:48.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.103+0000 7f93a3fff640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T10:44:48.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.103+0000 7f93aaffd640 10 monclient: get_auth_request con 0x7f938c06de90 auth_method 0 2024-03-29T10:44:48.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.103+0000 7f93a3fff640 10 client.4472.objecter ms_handle_connect 0x7f938c06de90 2024-03-29T10:44:48.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.110+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f939c050680 con 0x7f93ac0b17d0 2024-03-29T10:44:48.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.110+0000 7f93a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:48.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.110+0000 7f93a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:48.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.464+0000 7f93b21e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:48.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.464+0000 7f93b21e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:48.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.464+0000 7f93b21e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:48.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.464+0000 7f93b21e6640 1 -- 172.21.15.114:0/4279707016 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9390005330 con 0x7f93ac0b17d0 2024-03-29T10:44:48.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.465+0000 7f93a3fff640 1 -- 172.21.15.114:0/4279707016 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f939c0525b0 con 0x7f93ac0b17d0 2024-03-29T10:44:48.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.465+0000 7f93a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:48.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.465+0000 7f93a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:48.449 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:44:48.449 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:48.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.468+0000 7f93b21e6640 10 librados: watch_flush enter 2024-03-29T10:44:48.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.469+0000 7f93b21e6640 10 librados: watch_flush exit 2024-03-29T10:44:48.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.469+0000 7f93b21e6640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T10:44:48.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.469+0000 7f93b21e6640 10 client.4472.objecter successfully canceled tick 2024-03-29T10:44:48.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.469+0000 7f93b21e6640 1 -- 172.21.15.114:0/4279707016 >> v1:172.21.15.114:6816/775044307 conn(0x7f938c06de90 legacy=0x7f938c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:48.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.469+0000 7f93b21e6640 10 monclient: shutdown 2024-03-29T10:44:48.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.469+0000 7f93b21e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:48.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.469+0000 7f93b21e6640 1 -- 172.21.15.114:0/4279707016 >> v1:172.21.15.114:6790/0 conn(0x7f93ac0b17d0 legacy=0x7f93ac1323d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:48.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.469+0000 7f93b21e6640 1 -- 172.21.15.114:0/4279707016 shutdown_connections 2024-03-29T10:44:48.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.469+0000 7f93b21e6640 1 -- 172.21.15.114:0/4279707016 >> 172.21.15.114:0/4279707016 conn(0x7f93ac066a00 msgr2=0x7f93ac0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:48.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.469+0000 7f93b21e6640 1 -- 172.21.15.114:0/4279707016 shutdown_connections 2024-03-29T10:44:48.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.470+0000 7f93b21e6640 1 -- 172.21.15.114:0/4279707016 wait complete. 2024-03-29T10:44:48.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:48.470+0000 7f93b21e6640 1 librados: shutdown 2024-03-29T10:44:48.467 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:55.274 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:55.423 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.438+0000 7f50adfc4640 10 monclient: get_monmap_and_config 2024-03-29T10:44:55.423 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.438+0000 7f50adfc4640 10 monclient: build_initial_monmap 2024-03-29T10:44:55.423 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.438+0000 7f50adfc4640 10 monclient: monmap: 2024-03-29T10:44:55.423 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:55.423 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:55.423 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:44:55.440173+0000 2024-03-29T10:44:55.423 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:44:55.440173+0000 2024-03-29T10:44:55.424 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:55.424 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:55.424 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:55.424 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:44:55.424 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:44:55.424 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:55.424 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 1 Processor -- start 2024-03-29T10:44:55.424 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 1 -- start start 2024-03-29T10:44:55.424 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 10 monclient: init 2024-03-29T10:44:55.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:55.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:55.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 10 monclient(hunting): picked mon.noname-c con 0x7f50a80ae250 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:55.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 10 monclient(hunting): picked mon.noname-b con 0x7f50a80a9720 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:55.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 10 monclient(hunting): picked mon.noname-a con 0x7f50a814bba0 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:55.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a80a9b80 con 0x7f50a814bba0 2024-03-29T10:44:55.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a80aa7b0 con 0x7f50a80ae250 2024-03-29T10:44:55.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a80afbf0 con 0x7f50a80a9720 2024-03-29T10:44:55.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:55.425 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.440+0000 7f50adfc4640 10 monclient(hunting): authenticate will time out at 1160.681968s 2024-03-29T10:44:55.427 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.443+0000 7f50a7fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f50a814bba0 0x7f50a814df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46735/0 (socket says 172.21.15.114:46735) 2024-03-29T10:44:55.427 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.443+0000 7f50a7fff640 1 -- 172.21.15.114:0/2187340575 learned_addr learned my addr 172.21.15.114:0/2187340575 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:55.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 1 -- 172.21.15.114:0/2187340575 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725611233 0 0) 0x7f50a80afbf0 con 0x7f50a80a9720 2024-03-29T10:44:55.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:55.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:55.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:55.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:44:55.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 1 -- 172.21.15.114:0/2187340575 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5094003520 con 0x7f50a80a9720 2024-03-29T10:44:55.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 1 -- 172.21.15.114:0/2187340575 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611858179 0 0) 0x7f50a80a9b80 con 0x7f50a814bba0 2024-03-29T10:44:55.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:55.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:55.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:55.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:44:55.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.444+0000 7f50a67fc640 1 -- 172.21.15.114:0/2187340575 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50a80afbf0 con 0x7f50a814bba0 2024-03-29T10:44:55.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 1 -- 172.21.15.114:0/2187340575 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4279789298 0 0) 0x7f5094003520 con 0x7f50a80a9720 2024-03-29T10:44:55.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 1 -- 172.21.15.114:0/2187340575 >> v1:172.21.15.176:6789/0 conn(0x7f50a80ae250 legacy=0x7f50a8142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:55.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 1 -- 172.21.15.114:0/2187340575 >> v1:172.21.15.114:6789/0 conn(0x7f50a814bba0 legacy=0x7f50a814df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:55.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:55.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 1 monclient: found mon.noname-b 2024-03-29T10:44:55.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:55.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:44:55.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 1 -- 172.21.15.114:0/2187340575 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a8143280 con 0x7f50a80a9720 2024-03-29T10:44:55.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:55.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:55.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:55.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50adfc4640 5 monclient: authenticate success, global_id 4475 2024-03-29T10:44:55.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50adfc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:55.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 1 -- 172.21.15.114:0/2187340575 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f5098001a10 con 0x7f50a80a9720 2024-03-29T10:44:55.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:55.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:55.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50a67fc640 10 monclient: dump: 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.445+0000 7f50adfc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:55.434 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.446+0000 7f50a67fc640 1 -- 172.21.15.114:0/2187340575 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5098001b70 con 0x7f50a80a9720 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.446+0000 7f50a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.446+0000 7f50adfc4640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.446+0000 7f50adfc4640 4 set_mon_vals no callback set 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.446+0000 7f50a67fc640 1 -- 172.21.15.114:0/2187340575 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f5098001f00 con 0x7f50a80a9720 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.446+0000 7f50a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.446+0000 7f50a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.446+0000 7f50a67fc640 10 monclient: dump: 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:55.435 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.447+0000 7f50adfc4640 10 monclient: shutdown 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.447+0000 7f50adfc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.447+0000 7f50adfc4640 1 -- 172.21.15.114:0/2187340575 >> v1:172.21.15.114:6790/0 conn(0x7f50a80a9720 legacy=0x7f50a813f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.447+0000 7f50adfc4640 1 -- 172.21.15.114:0/2187340575 shutdown_connections 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.447+0000 7f50adfc4640 1 -- 172.21.15.114:0/2187340575 >> 172.21.15.114:0/2187340575 conn(0x7f50a80b35e0 msgr2=0x7f50a80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.447+0000 7f50adfc4640 1 -- 172.21.15.114:0/2187340575 shutdown_connections 2024-03-29T10:44:55.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.447+0000 7f50adfc4640 1 -- 172.21.15.114:0/2187340575 wait complete. 2024-03-29T10:44:55.443 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.448+0000 7f50adfc4640 10 monclient: build_initial_monmap 2024-03-29T10:44:55.443 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.448+0000 7f50adfc4640 10 monclient: monmap: 2024-03-29T10:44:55.443 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:44:55.443 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:55.443 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:44:55.443 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:44:55.443 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:55.444 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:55.444 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:44:55.444 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:44:55.444 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:44:55.444 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:55.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.448+0000 7f50adfc4640 1 librados: starting msgr at 2024-03-29T10:44:55.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.448+0000 7f50adfc4640 1 librados: starting objecter 2024-03-29T10:44:55.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.449+0000 7f50adfc4640 1 Processor -- start 2024-03-29T10:44:55.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.449+0000 7f50adfc4640 1 -- start start 2024-03-29T10:44:55.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.449+0000 7f50adfc4640 1 librados: setting wanted keys 2024-03-29T10:44:55.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.449+0000 7f50adfc4640 1 librados: calling monclient init 2024-03-29T10:44:55.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.449+0000 7f50adfc4640 10 monclient: init 2024-03-29T10:44:55.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.450+0000 7f50adfc4640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:55.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.450+0000 7f50adfc4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:55.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.450+0000 7f50adfc4640 10 monclient(hunting): picked mon.noname-b con 0x7f50a80a9720 addr v1:172.21.15.176:6789/0 2024-03-29T10:44:55.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.450+0000 7f50adfc4640 10 monclient(hunting): picked mon.noname-c con 0x7f50a80ae250 addr v1:172.21.15.114:6790/0 2024-03-29T10:44:55.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.450+0000 7f50adfc4640 10 monclient(hunting): picked mon.noname-a con 0x7f50a814bba0 addr v1:172.21.15.114:6789/0 2024-03-29T10:44:55.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.451+0000 7f50adfc4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a80ab310 con 0x7f50a814bba0 2024-03-29T10:44:55.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.451+0000 7f50adfc4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a8139910 con 0x7f50a80a9720 2024-03-29T10:44:55.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.451+0000 7f50adfc4640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a80ab560 con 0x7f50a80ae250 2024-03-29T10:44:55.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.451+0000 7f50adfc4640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:55.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.451+0000 7f50adfc4640 10 monclient(hunting): authenticate will time out at 1160.692497s 2024-03-29T10:44:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.451+0000 7f50a6ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f50a80ae250 0x7f50a813a060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:60354/0 (socket says 172.21.15.114:60354) 2024-03-29T10:44:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.451+0000 7f50a6ffd640 1 -- 172.21.15.114:0/1361304888 learned_addr learned my addr 172.21.15.114:0/1361304888 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:44:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.452+0000 7f509ffff640 10 client.?.objecter ms_handle_connect 0x7f50a80ae250 2024-03-29T10:44:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.452+0000 7f509ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587856727 0 0) 0x7f50a80ab560 con 0x7f50a80ae250 2024-03-29T10:44:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:44:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5088004eb0 con 0x7f50a80ae250 2024-03-29T10:44:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2643280813 0 0) 0x7f5088004eb0 con 0x7f50a80ae250 2024-03-29T10:44:55.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50a80ab560 con 0x7f50a80ae250 2024-03-29T10:44:55.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f5098001f40 con 0x7f50a80ae250 2024-03-29T10:44:55.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:55.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:55.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 10 monclient(hunting): dump: 2024-03-29T10:44:55.447 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:55.447 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:55.447 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:55.447 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:55.447 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:55.447 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.453+0000 7f509ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.454+0000 7f509ffff640 10 client.?.objecter ms_handle_connect 0x7f50a80a9720 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.454+0000 7f509ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.454+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3746462632 0 0) 0x7f50a80ab560 con 0x7f50a80ae250 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.454+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 >> v1:172.21.15.176:6789/0 conn(0x7f50a80a9720 legacy=0x7f50a813aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.454+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 >> v1:172.21.15.114:6789/0 conn(0x7f50a814bba0 legacy=0x7f50a81e6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.454+0000 7f509ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.454+0000 7f509ffff640 1 monclient: found mon.c 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.454+0000 7f509ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:55.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.454+0000 7f509ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.454+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a80ad2c0 con 0x7f50a80ae250 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.454+0000 7f509ffff640 10 monclient: _finish_auth 0 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.455+0000 7f509ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.455+0000 7f509ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.456+0000 7f50adfc4640 5 monclient: authenticate success, global_id 4478 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.456+0000 7f50adfc4640 10 monclient: _renew_subs 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.456+0000 7f50adfc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.456+0000 7f50adfc4640 1 -- 172.21.15.114:0/1361304888 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a80aba80 con 0x7f50a80ae250 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.456+0000 7f50adfc4640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.456+0000 7f50adfc4640 10 monclient: _renew_subs 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.456+0000 7f50adfc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:55.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.456+0000 7f50adfc4640 1 -- 172.21.15.114:0/1361304888 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a813b7e0 con 0x7f50a80ae250 2024-03-29T10:44:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.456+0000 7f50adfc4640 1 librados: init done 2024-03-29T10:44:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.457+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5098003c20 con 0x7f50a80ae250 2024-03-29T10:44:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.457+0000 7f509ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.457+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f5098004030 con 0x7f50a80ae250 2024-03-29T10:44:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.458+0000 7f50a5ffb640 4 set_mon_vals no callback set 2024-03-29T10:44:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.458+0000 7f509dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.459+0000 7f509dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.459+0000 7f509dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.459+0000 7f509dffb640 1 -- 172.21.15.114:0/1361304888 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a80ab7a0 con 0x7f50a80ae250 2024-03-29T10:44:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.460+0000 7f509ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.460+0000 7f509ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.460+0000 7f509ffff640 10 monclient: dump: 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:44:55.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.460+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f5098004190 con 0x7f50a80ae250 2024-03-29T10:44:55.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.460+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f5098085040 con 0x7f50a80ae250 2024-03-29T10:44:55.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.460+0000 7f509ffff640 10 client.4478.objecter ms_dispatch 0x7f50a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:55.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.460+0000 7f509ffff640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:55.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.460+0000 7f509ffff640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:55.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.460+0000 7f509ffff640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-29T10:44:55.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.461+0000 7f50a77fe640 10 monclient: get_auth_request con 0x7f508806db80 auth_method 0 2024-03-29T10:44:55.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.462+0000 7f509ffff640 10 client.4478.objecter ms_handle_connect 0x7f508806db80 2024-03-29T10:44:55.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.470+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50a80ab7a0 con 0x7f50a80ae250 2024-03-29T10:44:55.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.470+0000 7f509ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:55.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.470+0000 7f509ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:55.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.800+0000 7f509dffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:55.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.801+0000 7f509dffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:55.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.801+0000 7f509dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:44:55.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.801+0000 7f509dffb640 1 -- 172.21.15.114:0/1361304888 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50a80ab7a0 con 0x7f50a80ae250 2024-03-29T10:44:55.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.803+0000 7f509ffff640 1 -- 172.21.15.114:0/1361304888 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f50a80ab7a0 con 0x7f50a80ae250 2024-03-29T10:44:55.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.803+0000 7f509ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:55.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.803+0000 7f509ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:55.787 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:44:55.787 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:55.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.807+0000 7f509dffb640 10 librados: watch_flush enter 2024-03-29T10:44:55.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.807+0000 7f509dffb640 10 librados: watch_flush exit 2024-03-29T10:44:55.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.807+0000 7f509dffb640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-29T10:44:55.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.807+0000 7f509dffb640 10 client.4478.objecter successfully canceled tick 2024-03-29T10:44:55.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.807+0000 7f509dffb640 1 -- 172.21.15.114:0/1361304888 >> v1:172.21.15.114:6816/775044307 conn(0x7f508806db80 legacy=0x7f5088070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:55.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.807+0000 7f509dffb640 10 monclient: shutdown 2024-03-29T10:44:55.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.807+0000 7f509dffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:55.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.807+0000 7f509dffb640 1 -- 172.21.15.114:0/1361304888 >> v1:172.21.15.114:6790/0 conn(0x7f50a80ae250 legacy=0x7f50a813a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:55.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.807+0000 7f509dffb640 1 -- 172.21.15.114:0/1361304888 shutdown_connections 2024-03-29T10:44:55.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.807+0000 7f509dffb640 1 -- 172.21.15.114:0/1361304888 >> 172.21.15.114:0/1361304888 conn(0x7f50a80b35e0 msgr2=0x7f50a80b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:55.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.808+0000 7f509dffb640 1 -- 172.21.15.114:0/1361304888 shutdown_connections 2024-03-29T10:44:55.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.808+0000 7f509dffb640 1 -- 172.21.15.114:0/1361304888 wait complete. 2024-03-29T10:44:55.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:44:55.808+0000 7f509dffb640 1 librados: shutdown 2024-03-29T10:44:55.808 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:02.615 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:02.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.783+0000 7f41c970f640 10 monclient: get_monmap_and_config 2024-03-29T10:45:02.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.783+0000 7f41c970f640 10 monclient: build_initial_monmap 2024-03-29T10:45:02.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.783+0000 7f41c970f640 10 monclient: monmap: 2024-03-29T10:45:02.767 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:02.767 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:02.767 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:45:02.785136+0000 2024-03-29T10:45:02.767 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:45:02.785136+0000 2024-03-29T10:45:02.767 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:02.767 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:02.767 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:02.768 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:45:02.768 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:45:02.768 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.787+0000 7f41c970f640 1 Processor -- start 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.787+0000 7f41c970f640 1 -- start start 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.787+0000 7f41c970f640 10 monclient: init 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.788+0000 7f41c970f640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.788+0000 7f41c970f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.788+0000 7f41c970f640 10 monclient(hunting): picked mon.noname-a con 0x7f41c406d9d0 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.788+0000 7f41c970f640 10 monclient(hunting): picked mon.noname-c con 0x7f41c40b07d0 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.788+0000 7f41c970f640 10 monclient(hunting): picked mon.noname-b con 0x7f41c40af8e0 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.788+0000 7f41c970f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c40aed10 con 0x7f41c406d9d0 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.788+0000 7f41c970f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c40b0c00 con 0x7f41c40b07d0 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.788+0000 7f41c970f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c40afd30 con 0x7f41c40af8e0 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.788+0000 7f41c970f640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:02.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.788+0000 7f41c970f640 10 monclient(hunting): authenticate will time out at 1168.029267s 2024-03-29T10:45:02.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.791+0000 7f41c37fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f41c40af8e0 0x7f41c414a180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35226/0 (socket says 172.21.15.114:35226) 2024-03-29T10:45:02.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.791+0000 7f41c37fe640 1 -- 172.21.15.114:0/894184088 learned_addr learned my addr 172.21.15.114:0/894184088 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:02.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.792+0000 7f41c1ffb640 1 -- 172.21.15.114:0/894184088 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488354871 0 0) 0x7f41c40afd30 con 0x7f41c40af8e0 2024-03-29T10:45:02.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.792+0000 7f41c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:02.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.792+0000 7f41c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:02.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.792+0000 7f41c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:02.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.792+0000 7f41c1ffb640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:45:02.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.792+0000 7f41c1ffb640 1 -- 172.21.15.114:0/894184088 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ac004630 con 0x7f41c40af8e0 2024-03-29T10:45:02.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 1 -- 172.21.15.114:0/894184088 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1813291751 0 0) 0x7f41ac004630 con 0x7f41c40af8e0 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 1 -- 172.21.15.114:0/894184088 >> v1:172.21.15.176:6789/0 conn(0x7f41c40b07d0 legacy=0x7f41c4146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 1 -- 172.21.15.114:0/894184088 >> v1:172.21.15.114:6789/0 conn(0x7f41c406d9d0 legacy=0x7f41c40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 1 monclient: found mon.noname-b 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 1 -- 172.21.15.114:0/894184088 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c40ab1b0 con 0x7f41c40af8e0 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 1 -- 172.21.15.114:0/894184088 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f41b8001690 con 0x7f41c40af8e0 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:02.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c1ffb640 10 monclient: dump: 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c970f640 5 monclient: authenticate success, global_id 4481 2024-03-29T10:45:02.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.793+0000 7f41c970f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.794+0000 7f41c1ffb640 1 -- 172.21.15.114:0/894184088 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41b80017f0 con 0x7f41c40af8e0 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.794+0000 7f41c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.794+0000 7f41c1ffb640 1 -- 172.21.15.114:0/894184088 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f41b8001c40 con 0x7f41c40af8e0 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.794+0000 7f41c970f640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.794+0000 7f41c970f640 4 set_mon_vals no callback set 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.794+0000 7f41c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.794+0000 7f41c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.794+0000 7f41c1ffb640 10 monclient: dump: 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:02.782 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.795+0000 7f41c970f640 10 monclient: shutdown 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.795+0000 7f41c970f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.795+0000 7f41c970f640 1 -- 172.21.15.114:0/894184088 >> v1:172.21.15.114:6790/0 conn(0x7f41c40af8e0 legacy=0x7f41c414a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.795+0000 7f41c970f640 1 -- 172.21.15.114:0/894184088 shutdown_connections 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.795+0000 7f41c970f640 1 -- 172.21.15.114:0/894184088 >> 172.21.15.114:0/894184088 conn(0x7f41c406b660 msgr2=0x7f41c406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.795+0000 7f41c970f640 1 -- 172.21.15.114:0/894184088 shutdown_connections 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.795+0000 7f41c970f640 1 -- 172.21.15.114:0/894184088 wait complete. 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.797+0000 7f41c970f640 10 monclient: build_initial_monmap 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.797+0000 7f41c970f640 10 monclient: monmap: 2024-03-29T10:45:02.783 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:02.784 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:02.784 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:45:02.784 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:45:02.784 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:02.784 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:02.784 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:02.784 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:45:02.784 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:45:02.784 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:02.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.797+0000 7f41c970f640 1 librados: starting msgr at 2024-03-29T10:45:02.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.797+0000 7f41c970f640 1 librados: starting objecter 2024-03-29T10:45:02.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.797+0000 7f41c970f640 1 Processor -- start 2024-03-29T10:45:02.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.797+0000 7f41c970f640 1 -- start start 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.798+0000 7f41c970f640 1 librados: setting wanted keys 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.798+0000 7f41c970f640 1 librados: calling monclient init 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.798+0000 7f41c970f640 10 monclient: init 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.798+0000 7f41c970f640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.799+0000 7f41c970f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.799+0000 7f41c970f640 10 monclient(hunting): picked mon.noname-c con 0x7f41c406d9d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.799+0000 7f41c2ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f41c406d9d0 0x7f41c4136210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:37240/0 (socket says 172.21.15.114:37240) 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.799+0000 7f41c2ffd640 1 -- 172.21.15.114:0/2966276244 learned_addr learned my addr 172.21.15.114:0/2966276244 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.799+0000 7f41ab7fe640 10 client.?.objecter ms_handle_connect 0x7f41c406d9d0 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.799+0000 7f41ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.799+0000 7f41c970f640 10 monclient(hunting): picked mon.noname-a con 0x7f41c40af8e0 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41c970f640 10 monclient(hunting): picked mon.noname-b con 0x7f41c40b07d0 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41c970f640 1 -- 172.21.15.114:0/2966276244 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c4138120 con 0x7f41c40af8e0 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41c970f640 1 -- 172.21.15.114:0/2966276244 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c4138390 con 0x7f41c40b07d0 2024-03-29T10:45:02.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41c970f640 1 -- 172.21.15.114:0/2966276244 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c4137060 con 0x7f41c406d9d0 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41c970f640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41c970f640 10 monclient(hunting): authenticate will time out at 1168.041250s 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615526885 0 0) 0x7f41c4137060 con 0x7f41c406d9d0 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41ab7fe640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.800+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b0004eb0 con 0x7f41c406d9d0 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.801+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 852190931 0 0) 0x7f41b0004eb0 con 0x7f41c406d9d0 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.801+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41c4137060 con 0x7f41c406d9d0 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.801+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f41bc001d10 con 0x7f41c406d9d0 2024-03-29T10:45:02.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.801+0000 7f41ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:02.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.801+0000 7f41ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:02.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.801+0000 7f41ab7fe640 10 monclient(hunting): dump: 2024-03-29T10:45:02.790 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:02.790 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:02.790 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:02.790 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:02.790 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:02.790 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:02.790 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:02.790 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:02.790 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.801+0000 7f41ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.801+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4012084683 0 0) 0x7f41c4137060 con 0x7f41c406d9d0 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.801+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 >> v1:172.21.15.176:6789/0 conn(0x7f41c40b07d0 legacy=0x7f41c412a700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 >> v1:172.21.15.114:6789/0 conn(0x7f41c40af8e0 legacy=0x7f41c40b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41ab7fe640 1 monclient: found mon.c 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c4137580 con 0x7f41c406d9d0 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41ab7fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:02.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41ab7fe640 10 client.?.objecter ms_handle_connect 0x7f41c40af8e0 2024-03-29T10:45:02.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41c970f640 5 monclient: authenticate success, global_id 4484 2024-03-29T10:45:02.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.803+0000 7f41c970f640 10 monclient: _renew_subs 2024-03-29T10:45:02.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41c970f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:02.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41c970f640 1 -- 172.21.15.114:0/2966276244 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c4071240 con 0x7f41c406d9d0 2024-03-29T10:45:02.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41c970f640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:02.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41c970f640 10 monclient: _renew_subs 2024-03-29T10:45:02.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41c970f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:02.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41c970f640 1 -- 172.21.15.114:0/2966276244 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c40b3060 con 0x7f41c406d9d0 2024-03-29T10:45:02.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41c970f640 1 librados: init done 2024-03-29T10:45:02.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41ab7fe640 10 client.4484.objecter resend_mon_ops 2024-03-29T10:45:02.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41bc001e70 con 0x7f41c406d9d0 2024-03-29T10:45:02.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:02.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f41bc003de0 con 0x7f41c406d9d0 2024-03-29T10:45:02.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.804+0000 7f41c0ff9640 4 set_mon_vals no callback set 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.805+0000 7f41ab7fe640 10 monclient: dump: 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:02.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.807+0000 7f41a97fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:02.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.807+0000 7f41a97fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:02.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.807+0000 7f41a97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:02.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.807+0000 7f41a97fa640 1 -- 172.21.15.114:0/2966276244 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419c005080 con 0x7f41c406d9d0 2024-03-29T10:45:02.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.807+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f41bc003f40 con 0x7f41c406d9d0 2024-03-29T10:45:02.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.808+0000 7f41c27fc640 10 monclient: get_auth_request con 0x7f41b006db80 auth_method 0 2024-03-29T10:45:02.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.808+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f41bc084ee0 con 0x7f41c406d9d0 2024-03-29T10:45:02.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.808+0000 7f41ab7fe640 10 client.4484.objecter ms_dispatch 0x7f41c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:02.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.808+0000 7f41ab7fe640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:02.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.808+0000 7f41ab7fe640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:02.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.808+0000 7f41ab7fe640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-29T10:45:02.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.808+0000 7f41ab7fe640 10 client.4484.objecter ms_handle_connect 0x7f41b006db80 2024-03-29T10:45:02.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.817+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41bc0507c0 con 0x7f41c406d9d0 2024-03-29T10:45:02.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.817+0000 7f41ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:02.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:02.817+0000 7f41ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:03.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.144+0000 7f41a97fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:03.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.144+0000 7f41a97fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:03.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.144+0000 7f41a97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:03.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.144+0000 7f41a97fa640 1 -- 172.21.15.114:0/2966276244 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f419c005330 con 0x7f41c406d9d0 2024-03-29T10:45:03.129 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.146+0000 7f41ab7fe640 1 -- 172.21.15.114:0/2966276244 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f41bc0526f0 con 0x7f41c406d9d0 2024-03-29T10:45:03.129 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.146+0000 7f41ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:03.129 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.146+0000 7f41ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:03.129 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:45:03.129 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:03.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.150+0000 7f41a97fa640 10 librados: watch_flush enter 2024-03-29T10:45:03.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.150+0000 7f41a97fa640 10 librados: watch_flush exit 2024-03-29T10:45:03.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.150+0000 7f41a97fa640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-29T10:45:03.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.150+0000 7f41a97fa640 10 client.4484.objecter successfully canceled tick 2024-03-29T10:45:03.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.150+0000 7f41a97fa640 1 -- 172.21.15.114:0/2966276244 >> v1:172.21.15.114:6816/775044307 conn(0x7f41b006db80 legacy=0x7f41b0070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:03.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.150+0000 7f41a97fa640 10 monclient: shutdown 2024-03-29T10:45:03.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.150+0000 7f41a97fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:03.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.150+0000 7f41a97fa640 1 -- 172.21.15.114:0/2966276244 >> v1:172.21.15.114:6790/0 conn(0x7f41c406d9d0 legacy=0x7f41c4136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:03.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.151+0000 7f41a97fa640 1 -- 172.21.15.114:0/2966276244 shutdown_connections 2024-03-29T10:45:03.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.151+0000 7f41a97fa640 1 -- 172.21.15.114:0/2966276244 >> 172.21.15.114:0/2966276244 conn(0x7f41c406b660 msgr2=0x7f41c413da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:03.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.151+0000 7f41a97fa640 1 -- 172.21.15.114:0/2966276244 shutdown_connections 2024-03-29T10:45:03.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.151+0000 7f41a97fa640 1 -- 172.21.15.114:0/2966276244 wait complete. 2024-03-29T10:45:03.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:03.151+0000 7f41a97fa640 1 librados: shutdown 2024-03-29T10:45:03.151 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:09.958 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:10.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.125+0000 7f7ccb24a640 10 monclient: get_monmap_and_config 2024-03-29T10:45:10.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.125+0000 7f7ccb24a640 10 monclient: build_initial_monmap 2024-03-29T10:45:10.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.125+0000 7f7ccb24a640 10 monclient: monmap: 2024-03-29T10:45:10.108 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:10.108 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:10.108 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:45:10.126179+0000 2024-03-29T10:45:10.108 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:45:10.126179+0000 2024-03-29T10:45:10.108 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:10.108 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:10.108 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:10.109 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:45:10.109 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:45:10.109 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:10.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.129+0000 7f7ccb24a640 1 Processor -- start 2024-03-29T10:45:10.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.129+0000 7f7ccb24a640 1 -- start start 2024-03-29T10:45:10.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.129+0000 7f7ccb24a640 10 monclient: init 2024-03-29T10:45:10.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7ccb24a640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:10.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7ccb24a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:10.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7ccb24a640 10 monclient(hunting): picked mon.noname-b con 0x7f7cc40b4330 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:10.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7ccb24a640 10 monclient(hunting): picked mon.noname-a con 0x7f7cc41470b0 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:10.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7ccb24a640 10 monclient(hunting): picked mon.noname-c con 0x7f7cc41439b0 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:10.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7ccb24a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc406f2a0 con 0x7f7cc41470b0 2024-03-29T10:45:10.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7ccb24a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc4146b90 con 0x7f7cc41439b0 2024-03-29T10:45:10.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7ccb24a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc40b2ca0 con 0x7f7cc40b4330 2024-03-29T10:45:10.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7ccb24a640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:10.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7ccb24a640 10 monclient(hunting): authenticate will time out at 1175.370508s 2024-03-29T10:45:10.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7cc8fe6640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f7cc40b4330 0x7f7cc40b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:32946/0 (socket says 172.21.15.114:32946) 2024-03-29T10:45:10.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.130+0000 7f7cc8fe6640 1 -- 172.21.15.114:0/1629196276 learned_addr learned my addr 172.21.15.114:0/1629196276 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:10.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160891151 0 0) 0x7f7cc40b2ca0 con 0x7f7cc40b4330 2024-03-29T10:45:10.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:10.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:10.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:10.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:45:10.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb4003520 con 0x7f7cc40b4330 2024-03-29T10:45:10.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770044371 0 0) 0x7f7cc406f2a0 con 0x7f7cc41470b0 2024-03-29T10:45:10.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:10.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:10.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:10.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:45:10.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.134+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc40b2ca0 con 0x7f7cc41470b0 2024-03-29T10:45:10.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474688033 0 0) 0x7f7cc4146b90 con 0x7f7cc41439b0 2024-03-29T10:45:10.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:10.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:10.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:10.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:45:10.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc406f2a0 con 0x7f7cc41439b0 2024-03-29T10:45:10.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2241464890 0 0) 0x7f7cb4003520 con 0x7f7cc40b4330 2024-03-29T10:45:10.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 >> v1:172.21.15.176:6789/0 conn(0x7f7cc41439b0 legacy=0x7f7cc4151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:10.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 >> v1:172.21.15.114:6789/0 conn(0x7f7cc41470b0 legacy=0x7f7cc40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 1 monclient: found mon.noname-b 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc406e0c0 con 0x7f7cc40b4330 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7ccb24a640 5 monclient: authenticate success, global_id 4487 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7ccb24a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7cb0001610 con 0x7f7cc40b4330 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:10.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient: dump: 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7ccb24a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 1 -- 172.21.15.114:0/1629196276 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cb0001860 con 0x7f7cc40b4330 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7cc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7ccb24a640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:10.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.135+0000 7f7ccb24a640 4 set_mon_vals no callback set 2024-03-29T10:45:10.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.136+0000 7f7ccb24a640 10 monclient: shutdown 2024-03-29T10:45:10.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.136+0000 7f7ccb24a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:10.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.136+0000 7f7ccb24a640 1 -- 172.21.15.114:0/1629196276 >> v1:172.21.15.114:6790/0 conn(0x7f7cc40b4330 legacy=0x7f7cc40b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:10.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.136+0000 7f7ccb24a640 1 -- 172.21.15.114:0/1629196276 shutdown_connections 2024-03-29T10:45:10.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.136+0000 7f7ccb24a640 1 -- 172.21.15.114:0/1629196276 >> 172.21.15.114:0/1629196276 conn(0x7f7cc4066a00 msgr2=0x7f7cc4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:10.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.139+0000 7f7ccb24a640 1 -- 172.21.15.114:0/1629196276 shutdown_connections 2024-03-29T10:45:10.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.139+0000 7f7ccb24a640 1 -- 172.21.15.114:0/1629196276 wait complete. 2024-03-29T10:45:10.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.140+0000 7f7ccb24a640 10 monclient: build_initial_monmap 2024-03-29T10:45:10.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.140+0000 7f7ccb24a640 10 monclient: monmap: 2024-03-29T10:45:10.123 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:10.123 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:10.123 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:45:10.123 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:45:10.123 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:10.123 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:10.123 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:10.123 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:45:10.124 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:45:10.124 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:10.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.140+0000 7f7ccb24a640 1 librados: starting msgr at 2024-03-29T10:45:10.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.140+0000 7f7ccb24a640 1 librados: starting objecter 2024-03-29T10:45:10.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.140+0000 7f7ccb24a640 1 Processor -- start 2024-03-29T10:45:10.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.141+0000 7f7ccb24a640 1 -- start start 2024-03-29T10:45:10.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.141+0000 7f7ccb24a640 1 librados: setting wanted keys 2024-03-29T10:45:10.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.141+0000 7f7ccb24a640 1 librados: calling monclient init 2024-03-29T10:45:10.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.141+0000 7f7ccb24a640 10 monclient: init 2024-03-29T10:45:10.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.142+0000 7f7ccb24a640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:10.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.142+0000 7f7ccb24a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:10.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.142+0000 7f7ccb24a640 10 monclient(hunting): picked mon.noname-c con 0x7f7cc40b4330 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:10.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.142+0000 7f7ccb24a640 10 monclient(hunting): picked mon.noname-b con 0x7f7cc41439b0 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:10.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.142+0000 7f7ccb24a640 10 monclient(hunting): picked mon.noname-a con 0x7f7cc41470b0 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:10.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.142+0000 7f7ccb24a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc41df970 con 0x7f7cc41470b0 2024-03-29T10:45:10.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.142+0000 7f7ccb24a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc41df6b0 con 0x7f7cc41439b0 2024-03-29T10:45:10.128 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.142+0000 7f7ccb24a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc41276c0 con 0x7f7cc40b4330 2024-03-29T10:45:10.128 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.142+0000 7f7ccb24a640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:10.128 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.142+0000 7f7ccb24a640 10 monclient(hunting): authenticate will time out at 1175.382202s 2024-03-29T10:45:10.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc8fe6640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f7cc40b4330 0x7f7cc41defa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34818/0 (socket says 172.21.15.114:34818) 2024-03-29T10:45:10.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc8fe6640 1 -- 172.21.15.114:0/2722798106 learned_addr learned my addr 172.21.15.114:0/2722798106 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:10.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 10 client.?.objecter ms_handle_connect 0x7f7cc40b4330 2024-03-29T10:45:10.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:10.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110183718 0 0) 0x7f7cc41276c0 con 0x7f7cc40b4330 2024-03-29T10:45:10.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:10.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:10.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:10.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:45:10.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ca8004eb0 con 0x7f7cc40b4330 2024-03-29T10:45:10.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2596934473 0 0) 0x7f7ca8004eb0 con 0x7f7cc40b4330 2024-03-29T10:45:10.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cc41276c0 con 0x7f7cc40b4330 2024-03-29T10:45:10.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7cb0004980 con 0x7f7cc40b4330 2024-03-29T10:45:10.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:10.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:10.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.146+0000 7f7cc17fa640 10 monclient(hunting): dump: 2024-03-29T10:45:10.132 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.147+0000 7f7cc17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.147+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1314677075 0 0) 0x7f7cc41276c0 con 0x7f7cc40b4330 2024-03-29T10:45:10.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.147+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 >> v1:172.21.15.176:6789/0 conn(0x7f7cc41439b0 legacy=0x7f7cc4125340 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:45:10.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.147+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 >> v1:172.21.15.114:6789/0 conn(0x7f7cc41470b0 legacy=0x7f7cc4125a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:10.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.148+0000 7f7cc17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:45:10.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.148+0000 7f7cc17fa640 1 monclient: found mon.c 2024-03-29T10:45:10.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.148+0000 7f7cc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:10.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.148+0000 7f7cc17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:10.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.148+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc4141110 con 0x7f7cc40b4330 2024-03-29T10:45:10.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.148+0000 7f7cc17fa640 10 monclient: _finish_auth 0 2024-03-29T10:45:10.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.148+0000 7f7cc17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:45:10.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.148+0000 7f7cc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:10.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cb0004ec0 con 0x7f7cc40b4330 2024-03-29T10:45:10.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7cc17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:10.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7ccb24a640 5 monclient: authenticate success, global_id 4490 2024-03-29T10:45:10.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7ccb24a640 10 monclient: _renew_subs 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7ccb24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7ccb24a640 1 -- 172.21.15.114:0/2722798106 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc4141300 con 0x7f7cc40b4330 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7cc2ffd640 4 set_mon_vals no callback set 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7ccb24a640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7ccb24a640 10 monclient: _renew_subs 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7ccb24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7ccb24a640 1 -- 172.21.15.114:0/2722798106 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc40a96c0 con 0x7f7cc40b4330 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.149+0000 7f7ccb24a640 1 librados: init done 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.150+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7cb0001bf0 con 0x7f7cc40b4330 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.150+0000 7f7cc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.150+0000 7f7cc17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:10.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.150+0000 7f7cc17fa640 10 monclient: dump: 2024-03-29T10:45:10.140 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:10.140 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:10.140 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:10.140 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:10.140 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:10.140 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:10.140 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:10.140 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:10.140 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:10.140 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:10.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.151+0000 7f7ccb24a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:10.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.151+0000 7f7ccb24a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:10.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.151+0000 7f7ccb24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:10.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.151+0000 7f7ccb24a640 1 -- 172.21.15.114:0/2722798106 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc4075f10 con 0x7f7cc40b4330 2024-03-29T10:45:10.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.152+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f7cb00050b0 con 0x7f7cc40b4330 2024-03-29T10:45:10.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.152+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f7cb0084e60 con 0x7f7cc40b4330 2024-03-29T10:45:10.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.152+0000 7f7cc17fa640 10 client.4490.objecter ms_dispatch 0x7f7cc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:10.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.152+0000 7f7cc17fa640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:10.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.152+0000 7f7cc17fa640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:10.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.152+0000 7f7cc17fa640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-29T10:45:10.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.153+0000 7f7cc3fff640 10 monclient: get_auth_request con 0x7f7ca806dae0 auth_method 0 2024-03-29T10:45:10.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.153+0000 7f7cc17fa640 10 client.4490.objecter ms_handle_connect 0x7f7ca806dae0 2024-03-29T10:45:10.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.163+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7cb0050740 con 0x7f7cc40b4330 2024-03-29T10:45:10.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.163+0000 7f7cc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:10.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.163+0000 7f7cc17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:10.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.482+0000 7f7ccb24a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:10.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.482+0000 7f7ccb24a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:10.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.482+0000 7f7ccb24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:10.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.482+0000 7f7ccb24a640 1 -- 172.21.15.114:0/2722798106 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cc4067110 con 0x7f7cc40b4330 2024-03-29T10:45:10.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.483+0000 7f7cc17fa640 1 -- 172.21.15.114:0/2722798106 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f7cb0052670 con 0x7f7cc40b4330 2024-03-29T10:45:10.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.483+0000 7f7cc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:10.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.483+0000 7f7cc17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:10.465 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:45:10.465 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:10.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.486+0000 7f7ccb24a640 10 librados: watch_flush enter 2024-03-29T10:45:10.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.487+0000 7f7ccb24a640 10 librados: watch_flush exit 2024-03-29T10:45:10.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.487+0000 7f7ccb24a640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-29T10:45:10.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.487+0000 7f7ccb24a640 10 client.4490.objecter successfully canceled tick 2024-03-29T10:45:10.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.487+0000 7f7ccb24a640 1 -- 172.21.15.114:0/2722798106 >> v1:172.21.15.114:6816/775044307 conn(0x7f7ca806dae0 legacy=0x7f7ca806ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:10.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.487+0000 7f7ccb24a640 10 monclient: shutdown 2024-03-29T10:45:10.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.487+0000 7f7ccb24a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:10.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.487+0000 7f7ccb24a640 1 -- 172.21.15.114:0/2722798106 >> v1:172.21.15.114:6790/0 conn(0x7f7cc40b4330 legacy=0x7f7cc41defa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:10.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.487+0000 7f7ccb24a640 1 -- 172.21.15.114:0/2722798106 shutdown_connections 2024-03-29T10:45:10.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.487+0000 7f7ccb24a640 1 -- 172.21.15.114:0/2722798106 >> 172.21.15.114:0/2722798106 conn(0x7f7cc4066a00 msgr2=0x7f7cc40b71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:10.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.488+0000 7f7ccb24a640 1 -- 172.21.15.114:0/2722798106 shutdown_connections 2024-03-29T10:45:10.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.488+0000 7f7ccb24a640 1 -- 172.21.15.114:0/2722798106 wait complete. 2024-03-29T10:45:10.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:10.488+0000 7f7ccb24a640 1 librados: shutdown 2024-03-29T10:45:10.487 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:17.294 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:17.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.466+0000 7fc684a4e640 10 monclient: get_monmap_and_config 2024-03-29T10:45:17.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.466+0000 7fc684a4e640 10 monclient: build_initial_monmap 2024-03-29T10:45:17.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.466+0000 7fc684a4e640 10 monclient: monmap: 2024-03-29T10:45:17.449 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:17.449 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:17.449 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:45:17.467407+0000 2024-03-29T10:45:17.449 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:45:17.467407+0000 2024-03-29T10:45:17.449 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:17.449 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:17.449 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:17.450 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:45:17.450 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:45:17.450 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:17.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.467+0000 7fc684a4e640 1 Processor -- start 2024-03-29T10:45:17.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.468+0000 7fc684a4e640 1 -- start start 2024-03-29T10:45:17.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.468+0000 7fc684a4e640 10 monclient: init 2024-03-29T10:45:17.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc684a4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:17.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc684a4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:17.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc684a4e640 10 monclient(hunting): picked mon.noname-b con 0x7fc6801457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:17.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc684a4e640 10 monclient(hunting): picked mon.noname-c con 0x7fc68013f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:17.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc67e59c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fc6801457f0 0x7fc68013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:53058/0 (socket says 172.21.15.114:53058) 2024-03-29T10:45:17.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc684a4e640 10 monclient(hunting): picked mon.noname-a con 0x7fc6800a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:17.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc684a4e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6800aaef0 con 0x7fc6800a9d70 2024-03-29T10:45:17.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc684a4e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc680145bd0 con 0x7fc68013f650 2024-03-29T10:45:17.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc684a4e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc680145eb0 con 0x7fc6801457f0 2024-03-29T10:45:17.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc684a4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:17.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc67e59c640 1 -- 172.21.15.114:0/3863607451 learned_addr learned my addr 172.21.15.114:0/3863607451 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:17.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.470+0000 7fc684a4e640 10 monclient(hunting): authenticate will time out at 1182.711786s 2024-03-29T10:45:17.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.471+0000 7fc67d59a640 1 -- 172.21.15.114:0/3863607451 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902236426 0 0) 0x7fc680145eb0 con 0x7fc6801457f0 2024-03-29T10:45:17.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.471+0000 7fc67d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:17.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.471+0000 7fc67d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:17.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.471+0000 7fc67d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:17.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.471+0000 7fc67d59a640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:45:17.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.471+0000 7fc67d59a640 1 -- 172.21.15.114:0/3863607451 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc668003520 con 0x7fc6801457f0 2024-03-29T10:45:17.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.472+0000 7fc67d59a640 1 -- 172.21.15.114:0/3863607451 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1306887031 0 0) 0x7fc668003520 con 0x7fc6801457f0 2024-03-29T10:45:17.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.472+0000 7fc67d59a640 1 -- 172.21.15.114:0/3863607451 >> v1:172.21.15.176:6789/0 conn(0x7fc68013f650 legacy=0x7fc6800b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:17.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.473+0000 7fc67d59a640 1 -- 172.21.15.114:0/3863607451 >> v1:172.21.15.114:6789/0 conn(0x7fc6800a9d70 legacy=0x7fc68014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.474+0000 7fc67d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 1 monclient: found mon.noname-b 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 1 -- 172.21.15.114:0/3863607451 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc680146340 con 0x7fc6801457f0 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 10 monclient: _finish_auth 0 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 1 -- 172.21.15.114:0/3863607451 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc674001690 con 0x7fc6801457f0 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 10 monclient: dump: 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:17.460 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 1 -- 172.21.15.114:0/3863607451 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc674001860 con 0x7fc6801457f0 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 1 -- 172.21.15.114:0/3863607451 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc674001cb0 con 0x7fc6801457f0 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:17.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc67d59a640 10 monclient: dump: 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc684a4e640 5 monclient: authenticate success, global_id 4493 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc684a4e640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.475+0000 7fc684a4e640 4 set_mon_vals no callback set 2024-03-29T10:45:17.462 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.476+0000 7fc684a4e640 10 monclient: shutdown 2024-03-29T10:45:17.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.476+0000 7fc684a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:17.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.476+0000 7fc684a4e640 1 -- 172.21.15.114:0/3863607451 >> v1:172.21.15.114:6790/0 conn(0x7fc6801457f0 legacy=0x7fc68013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:17.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.476+0000 7fc684a4e640 1 -- 172.21.15.114:0/3863607451 shutdown_connections 2024-03-29T10:45:17.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.476+0000 7fc684a4e640 1 -- 172.21.15.114:0/3863607451 >> 172.21.15.114:0/3863607451 conn(0x7fc680066a00 msgr2=0x7fc680066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:17.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.479+0000 7fc684a4e640 1 -- 172.21.15.114:0/3863607451 shutdown_connections 2024-03-29T10:45:17.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.479+0000 7fc684a4e640 1 -- 172.21.15.114:0/3863607451 wait complete. 2024-03-29T10:45:17.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.480+0000 7fc684a4e640 10 monclient: build_initial_monmap 2024-03-29T10:45:17.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.481+0000 7fc684a4e640 10 monclient: monmap: 2024-03-29T10:45:17.465 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:17.465 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:17.465 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:45:17.466 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:45:17.466 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:17.466 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:17.466 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:17.466 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:45:17.466 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:45:17.466 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:17.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.482+0000 7fc684a4e640 1 librados: starting msgr at 2024-03-29T10:45:17.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.482+0000 7fc684a4e640 1 librados: starting objecter 2024-03-29T10:45:17.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.482+0000 7fc684a4e640 1 Processor -- start 2024-03-29T10:45:17.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.484+0000 7fc684a4e640 1 -- start start 2024-03-29T10:45:17.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.484+0000 7fc684a4e640 1 librados: setting wanted keys 2024-03-29T10:45:17.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.484+0000 7fc684a4e640 1 librados: calling monclient init 2024-03-29T10:45:17.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.484+0000 7fc684a4e640 10 monclient: init 2024-03-29T10:45:17.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.485+0000 7fc684a4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:17.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.485+0000 7fc684a4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:17.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.485+0000 7fc684a4e640 10 monclient(hunting): picked mon.noname-a con 0x7fc6800a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:17.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.485+0000 7fc684a4e640 10 monclient(hunting): picked mon.noname-b con 0x7fc68013f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:17.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.485+0000 7fc684a4e640 10 monclient(hunting): picked mon.noname-c con 0x7fc68013ba80 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:17.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.485+0000 7fc684a4e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc680071c80 con 0x7fc6800a9d70 2024-03-29T10:45:17.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.485+0000 7fc684a4e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc680071e10 con 0x7fc68013f650 2024-03-29T10:45:17.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.485+0000 7fc684a4e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68006fd50 con 0x7fc68013ba80 2024-03-29T10:45:17.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.485+0000 7fc684a4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:17.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.485+0000 7fc684a4e640 10 monclient(hunting): authenticate will time out at 1182.726933s 2024-03-29T10:45:17.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.487+0000 7fc67ed9d640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fc68013ba80 0x7fc6801e7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55380/0 (socket says 172.21.15.114:55380) 2024-03-29T10:45:17.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.487+0000 7fc67ed9d640 1 -- 172.21.15.114:0/1103605203 learned_addr learned my addr 172.21.15.114:0/1103605203 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:17.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.487+0000 7fc66effd640 10 client.?.objecter ms_handle_connect 0x7fc68013ba80 2024-03-29T10:45:17.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.487+0000 7fc66effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:17.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.488+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617256762 0 0) 0x7fc68006fd50 con 0x7fc68013ba80 2024-03-29T10:45:17.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.488+0000 7fc66effd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:17.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.488+0000 7fc66effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:17.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.488+0000 7fc66effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:17.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.488+0000 7fc66effd640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:45:17.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.488+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc660004eb0 con 0x7fc68013ba80 2024-03-29T10:45:17.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 client.?.objecter ms_handle_connect 0x7fc6800a9d70 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 918346249 0 0) 0x7fc660004eb0 con 0x7fc68013ba80 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc68006fd50 con 0x7fc68013ba80 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc678001c60 con 0x7fc68013ba80 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): dump: 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:17.475 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945848148 0 0) 0x7fc680071c80 con 0x7fc6800a9d70 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): my global_id is 4449 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc660004eb0 con 0x7fc6800a9d70 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 client.?.objecter ms_handle_connect 0x7fc68013f650 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219293749 0 0) 0x7fc670000ff0 con 0x7fc68013f650 2024-03-29T10:45:17.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc680071c80 con 0x7fc68013f650 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 827220420 0 0) 0x7fc68006fd50 con 0x7fc68013ba80 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 >> v1:172.21.15.176:6789/0 conn(0x7fc68013f650 legacy=0x7fc6801df700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 >> v1:172.21.15.114:6789/0 conn(0x7fc6800a9d70 legacy=0x7fc68013b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 1 monclient: found mon.c 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.489+0000 7fc66effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc66effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68006fee0 con 0x7fc68013ba80 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc66effd640 10 monclient: _finish_auth 0 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc66effd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:17.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc66effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc678001e30 con 0x7fc68013ba80 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc66effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fc678003c20 con 0x7fc68013ba80 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc66effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc66effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc66effd640 10 monclient: dump: 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:17.478 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc684a4e640 5 monclient: authenticate success, global_id 4496 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc684a4e640 10 monclient: _renew_subs 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc684a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc684a4e640 1 -- 172.21.15.114:0/1103605203 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc680070830 con 0x7fc68013ba80 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc684a4e640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc684a4e640 10 monclient: _renew_subs 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc684a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc684a4e640 1 -- 172.21.15.114:0/1103605203 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc680071740 con 0x7fc68013ba80 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.490+0000 7fc684a4e640 1 librados: init done 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.491+0000 7fc66ffff640 4 set_mon_vals no callback set 2024-03-29T10:45:17.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.492+0000 7fc684a4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:17.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.492+0000 7fc684a4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:17.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.492+0000 7fc684a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:17.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.492+0000 7fc684a4e640 1 -- 172.21.15.114:0/1103605203 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc664005080 con 0x7fc68013ba80 2024-03-29T10:45:17.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.495+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fc678003d80 con 0x7fc68013ba80 2024-03-29T10:45:17.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.497+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fc678084e00 con 0x7fc68013ba80 2024-03-29T10:45:17.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.497+0000 7fc66effd640 10 client.4496.objecter ms_dispatch 0x7fc680000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:17.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.497+0000 7fc66effd640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:17.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.497+0000 7fc66effd640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:17.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.497+0000 7fc66effd640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T10:45:17.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.498+0000 7fc67e59c640 10 monclient: get_auth_request con 0x7fc66006de60 auth_method 0 2024-03-29T10:45:17.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.498+0000 7fc66effd640 10 client.4496.objecter ms_handle_connect 0x7fc66006de60 2024-03-29T10:45:17.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.504+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc678050510 con 0x7fc68013ba80 2024-03-29T10:45:17.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.504+0000 7fc66effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:17.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.504+0000 7fc66effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:17.823 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:45:17.824 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:17.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.839+0000 7fc684a4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:17.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.839+0000 7fc684a4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:17.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.839+0000 7fc684a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:17.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.839+0000 7fc684a4e640 1 -- 172.21.15.114:0/1103605203 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc664005330 con 0x7fc68013ba80 2024-03-29T10:45:17.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.839+0000 7fc66effd640 1 -- 172.21.15.114:0/1103605203 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fc678052610 con 0x7fc68013ba80 2024-03-29T10:45:17.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.839+0000 7fc66effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:17.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.839+0000 7fc66effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:17.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.843+0000 7fc684a4e640 10 librados: watch_flush enter 2024-03-29T10:45:17.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.843+0000 7fc684a4e640 10 librados: watch_flush exit 2024-03-29T10:45:17.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.843+0000 7fc684a4e640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T10:45:17.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.843+0000 7fc684a4e640 10 client.4496.objecter successfully canceled tick 2024-03-29T10:45:17.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.844+0000 7fc684a4e640 1 -- 172.21.15.114:0/1103605203 >> v1:172.21.15.114:6816/775044307 conn(0x7fc66006de60 legacy=0x7fc660070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:17.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.844+0000 7fc684a4e640 10 monclient: shutdown 2024-03-29T10:45:17.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.844+0000 7fc684a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:17.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.844+0000 7fc684a4e640 1 -- 172.21.15.114:0/1103605203 >> v1:172.21.15.114:6790/0 conn(0x7fc68013ba80 legacy=0x7fc6801e7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:17.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.844+0000 7fc684a4e640 1 -- 172.21.15.114:0/1103605203 shutdown_connections 2024-03-29T10:45:17.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.844+0000 7fc684a4e640 1 -- 172.21.15.114:0/1103605203 >> 172.21.15.114:0/1103605203 conn(0x7fc680066a00 msgr2=0x7fc6800b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:17.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.844+0000 7fc684a4e640 1 -- 172.21.15.114:0/1103605203 shutdown_connections 2024-03-29T10:45:17.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.844+0000 7fc684a4e640 1 -- 172.21.15.114:0/1103605203 wait complete. 2024-03-29T10:45:17.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:17.844+0000 7fc684a4e640 1 librados: shutdown 2024-03-29T10:45:17.844 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:24.651 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:24.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.815+0000 7f715b18c640 10 monclient: get_monmap_and_config 2024-03-29T10:45:24.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.815+0000 7f715b18c640 10 monclient: build_initial_monmap 2024-03-29T10:45:24.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.816+0000 7f715b18c640 10 monclient: monmap: 2024-03-29T10:45:24.799 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:24.800 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:24.800 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:45:24.817252+0000 2024-03-29T10:45:24.800 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:45:24.817252+0000 2024-03-29T10:45:24.800 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:24.800 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:24.800 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:24.800 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:45:24.800 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:45:24.800 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:24.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.817+0000 7f715b18c640 1 Processor -- start 2024-03-29T10:45:24.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.817+0000 7f715b18c640 1 -- start start 2024-03-29T10:45:24.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.817+0000 7f715b18c640 10 monclient: init 2024-03-29T10:45:24.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.818+0000 7f715b18c640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:24.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.818+0000 7f715b18c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:24.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.818+0000 7f715b18c640 10 monclient(hunting): picked mon.noname-b con 0x7f71540a9ac0 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:24.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.818+0000 7f715b18c640 10 monclient(hunting): picked mon.noname-a con 0x7f715413ee90 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:24.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.818+0000 7f715b18c640 10 monclient(hunting): picked mon.noname-c con 0x7f71540af900 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:24.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.818+0000 7f715b18c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71540afd10 con 0x7f715413ee90 2024-03-29T10:45:24.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.818+0000 7f715b18c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71540af6f0 con 0x7f71540af900 2024-03-29T10:45:24.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.818+0000 7f715b18c640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71540aac80 con 0x7f71540a9ac0 2024-03-29T10:45:24.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.818+0000 7f715b18c640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:24.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.818+0000 7f715b18c640 10 monclient(hunting): authenticate will time out at 1190.059329s 2024-03-29T10:45:24.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.821+0000 7f7159729640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f71540af900 0x7f7154149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:37541/0 (socket says 172.21.15.114:37541) 2024-03-29T10:45:24.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.821+0000 7f7159729640 1 -- 172.21.15.114:0/755562324 learned_addr learned my addr 172.21.15.114:0/755562324 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:24.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.824+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118783423 0 0) 0x7f71540af6f0 con 0x7f71540af900 2024-03-29T10:45:24.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.824+0000 7f71537fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:24.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.824+0000 7f71537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:24.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.824+0000 7f71537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:24.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.824+0000 7f71537fe640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:45:24.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.824+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713c004630 con 0x7f71540af900 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.825+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792435221 0 0) 0x7f71540afd10 con 0x7f715413ee90 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.825+0000 7f71537fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.825+0000 7f71537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.825+0000 7f71537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.825+0000 7f71537fe640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71540af6f0 con 0x7f715413ee90 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137759854 0 0) 0x7f71540aac80 con 0x7f71540a9ac0 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71540afd10 con 0x7f71540a9ac0 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1583143061 0 0) 0x7f713c004630 con 0x7f71540af900 2024-03-29T10:45:24.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 >> v1:172.21.15.114:6790/0 conn(0x7f71540a9ac0 legacy=0x7f71540adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 >> v1:172.21.15.114:6789/0 conn(0x7f715413ee90 legacy=0x7f715413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 1 monclient: found mon.noname-c 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71540aaf60 con 0x7f71540af900 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.826+0000 7f71537fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f71537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f715b18c640 5 monclient: authenticate success, global_id 4426 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f715b18c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f714c005030 con 0x7f71540af900 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f71537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:24.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f71537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f71537fe640 10 monclient: dump: 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f715b18c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:24.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f71537fe640 1 -- 172.21.15.114:0/755562324 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f714c005250 con 0x7f71540af900 2024-03-29T10:45:24.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f71537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:24.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f715b18c640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:24.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.827+0000 7f715b18c640 4 set_mon_vals no callback set 2024-03-29T10:45:24.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.828+0000 7f715b18c640 10 monclient: shutdown 2024-03-29T10:45:24.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.828+0000 7f715b18c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:24.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.828+0000 7f715b18c640 1 -- 172.21.15.114:0/755562324 >> v1:172.21.15.176:6789/0 conn(0x7f71540af900 legacy=0x7f7154149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:24.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.829+0000 7f715b18c640 1 -- 172.21.15.114:0/755562324 shutdown_connections 2024-03-29T10:45:24.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.829+0000 7f715b18c640 1 -- 172.21.15.114:0/755562324 >> 172.21.15.114:0/755562324 conn(0x7f71540b35e0 msgr2=0x7f71540b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:24.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.831+0000 7f715b18c640 1 -- 172.21.15.114:0/755562324 shutdown_connections 2024-03-29T10:45:24.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.832+0000 7f715b18c640 1 -- 172.21.15.114:0/755562324 wait complete. 2024-03-29T10:45:24.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.833+0000 7f715b18c640 10 monclient: build_initial_monmap 2024-03-29T10:45:24.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.833+0000 7f715b18c640 10 monclient: monmap: 2024-03-29T10:45:24.816 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:24.816 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:24.816 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:45:24.817 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:45:24.817 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:24.817 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:24.817 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:24.817 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:45:24.817 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:45:24.817 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:24.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.833+0000 7f715b18c640 1 librados: starting msgr at 2024-03-29T10:45:24.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.833+0000 7f715b18c640 1 librados: starting objecter 2024-03-29T10:45:24.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 1 Processor -- start 2024-03-29T10:45:24.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 1 -- start start 2024-03-29T10:45:24.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 1 librados: setting wanted keys 2024-03-29T10:45:24.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 1 librados: calling monclient init 2024-03-29T10:45:24.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 10 monclient: init 2024-03-29T10:45:24.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:24.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:24.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 10 monclient(hunting): picked mon.noname-c con 0x7f71540a9ac0 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:24.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 10 monclient(hunting): picked mon.noname-b con 0x7f71540af900 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:24.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 10 monclient(hunting): picked mon.noname-a con 0x7f715413ee90 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:24.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715413a020 con 0x7f715413ee90 2024-03-29T10:45:24.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7154139c80 con 0x7f71540af900 2024-03-29T10:45:24.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715413ac50 con 0x7f71540a9ac0 2024-03-29T10:45:24.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:24.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.834+0000 7f715b18c640 10 monclient(hunting): authenticate will time out at 1190.075951s 2024-03-29T10:45:24.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.839+0000 7f7159729640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f715413ee90 0x7f71541ddc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43741/0 (socket says 172.21.15.114:43741) 2024-03-29T10:45:24.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.839+0000 7f7159729640 1 -- 172.21.15.114:0/3620600686 learned_addr learned my addr 172.21.15.114:0/3620600686 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:24.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.839+0000 7f71517fa640 10 client.?.objecter ms_handle_connect 0x7f71540a9ac0 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.839+0000 7f71517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.839+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287487473 0 0) 0x7f715413ac50 con 0x7f71540a9ac0 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.839+0000 7f71517fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.839+0000 7f71517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.839+0000 7f71517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.839+0000 7f71517fa640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.839+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7140004eb0 con 0x7f71540a9ac0 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3445880332 0 0) 0x7f7140004eb0 con 0x7f71540a9ac0 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7140005300 con 0x7f71540a9ac0 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 client.?.objecter ms_handle_connect 0x7f71540af900 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f71440018f0 con 0x7f71540a9ac0 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:24.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient(hunting): dump: 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:24.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:24.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 client.?.objecter ms_handle_connect 0x7f715413ee90 2024-03-29T10:45:24.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:24.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136429461 0 0) 0x7f7154139c80 con 0x7f71540af900 2024-03-29T10:45:24.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:24.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:24.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:24.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient(hunting): my global_id is 4429 2024-03-29T10:45:24.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71400058a0 con 0x7f71540af900 2024-03-29T10:45:24.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3852744960 0 0) 0x7f7140005300 con 0x7f71540a9ac0 2024-03-29T10:45:24.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 >> v1:172.21.15.176:6789/0 conn(0x7f71540af900 legacy=0x7f715413a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:24.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 >> v1:172.21.15.114:6789/0 conn(0x7f715413ee90 legacy=0x7f71541ddc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:24.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient: _finish_hunting 0 2024-03-29T10:45:24.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 1 monclient: found mon.c 2024-03-29T10:45:24.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:24.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:24.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715413ade0 con 0x7f71540a9ac0 2024-03-29T10:45:24.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient: _finish_auth 0 2024-03-29T10:45:24.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 10 monclient: _check_auth_tickets 2024-03-29T10:45:24.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.840+0000 7f71517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:24.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.842+0000 7f715b18c640 5 monclient: authenticate success, global_id 4502 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.842+0000 7f715b18c640 10 monclient: _renew_subs 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.842+0000 7f715b18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.842+0000 7f715b18c640 1 -- 172.21.15.114:0/3620600686 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71541c8c30 con 0x7f71540a9ac0 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.843+0000 7f715b18c640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.843+0000 7f715b18c640 10 monclient: _renew_subs 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.843+0000 7f715b18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.843+0000 7f715b18c640 1 -- 172.21.15.114:0/3620600686 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71541d6990 con 0x7f71540a9ac0 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.843+0000 7f715b18c640 1 librados: init done 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.844+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71440035f0 con 0x7f71540a9ac0 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.844+0000 7f71517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.844+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f71440018f0 con 0x7f71540a9ac0 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.844+0000 7f71517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:24.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.844+0000 7f71517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.844+0000 7f71517fa640 10 monclient: dump: 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:24.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.845+0000 7f7152ffd640 4 set_mon_vals no callback set 2024-03-29T10:45:24.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.846+0000 7f715b18c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:24.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.846+0000 7f715b18c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:24.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.846+0000 7f715b18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:24.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.846+0000 7f715b18c640 1 -- 172.21.15.114:0/3620600686 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7134005080 con 0x7f71540a9ac0 2024-03-29T10:45:24.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.852+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f7144019690 con 0x7f71540a9ac0 2024-03-29T10:45:24.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.852+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f7144083370 con 0x7f71540a9ac0 2024-03-29T10:45:24.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.852+0000 7f71517fa640 10 client.4502.objecter ms_dispatch 0x7f7154000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:24.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.852+0000 7f71517fa640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:24.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.852+0000 7f71517fa640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:24.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.852+0000 7f7153fff640 10 monclient: get_auth_request con 0x7f714006e330 auth_method 0 2024-03-29T10:45:24.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.852+0000 7f71517fa640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T10:45:24.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.853+0000 7f71517fa640 10 client.4502.objecter ms_handle_connect 0x7f714006e330 2024-03-29T10:45:24.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.857+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f714404ea80 con 0x7f71540a9ac0 2024-03-29T10:45:24.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.857+0000 7f71517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:24.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:24.857+0000 7f71517fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:25.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.189+0000 7f715b18c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:25.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.189+0000 7f715b18c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:25.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.189+0000 7f715b18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:25.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.189+0000 7f715b18c640 1 -- 172.21.15.114:0/3620600686 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7134005330 con 0x7f71540a9ac0 2024-03-29T10:45:25.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.191+0000 7f71517fa640 1 -- 172.21.15.114:0/3620600686 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f7144001750 con 0x7f71540a9ac0 2024-03-29T10:45:25.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.191+0000 7f71517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:25.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.191+0000 7f71517fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:25.175 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:45:25.175 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:25.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.197+0000 7f715b18c640 10 librados: watch_flush enter 2024-03-29T10:45:25.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.198+0000 7f715b18c640 10 librados: watch_flush exit 2024-03-29T10:45:25.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.198+0000 7f715b18c640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T10:45:25.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.198+0000 7f715b18c640 10 client.4502.objecter successfully canceled tick 2024-03-29T10:45:25.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.198+0000 7f715b18c640 1 -- 172.21.15.114:0/3620600686 >> v1:172.21.15.114:6816/775044307 conn(0x7f714006e330 legacy=0x7f71400707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:25.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.198+0000 7f715b18c640 10 monclient: shutdown 2024-03-29T10:45:25.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.198+0000 7f715b18c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:25.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.198+0000 7f715b18c640 1 -- 172.21.15.114:0/3620600686 >> v1:172.21.15.114:6790/0 conn(0x7f71540a9ac0 legacy=0x7f71541d6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:25.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.199+0000 7f715b18c640 1 -- 172.21.15.114:0/3620600686 shutdown_connections 2024-03-29T10:45:25.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.199+0000 7f715b18c640 1 -- 172.21.15.114:0/3620600686 >> 172.21.15.114:0/3620600686 conn(0x7f71540b35e0 msgr2=0x7f71540adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:25.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.204+0000 7f715b18c640 1 -- 172.21.15.114:0/3620600686 shutdown_connections 2024-03-29T10:45:25.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.204+0000 7f715b18c640 1 -- 172.21.15.114:0/3620600686 wait complete. 2024-03-29T10:45:25.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:25.204+0000 7f715b18c640 1 librados: shutdown 2024-03-29T10:45:25.204 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:32.011 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:32.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.174+0000 7fdc93fff640 10 monclient: get_monmap_and_config 2024-03-29T10:45:32.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.174+0000 7fdc93fff640 10 monclient: build_initial_monmap 2024-03-29T10:45:32.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.174+0000 7fdc93fff640 10 monclient: monmap: 2024-03-29T10:45:32.159 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:32.159 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:32.159 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:45:32.176206+0000 2024-03-29T10:45:32.159 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:45:32.176206+0000 2024-03-29T10:45:32.159 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:32.159 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:32.159 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:32.159 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:45:32.160 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:45:32.160 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:32.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.176+0000 7fdc93fff640 1 Processor -- start 2024-03-29T10:45:32.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.176+0000 7fdc93fff640 1 -- start start 2024-03-29T10:45:32.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.176+0000 7fdc93fff640 10 monclient: init 2024-03-29T10:45:32.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.177+0000 7fdc93fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:32.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.177+0000 7fdc93fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:32.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.177+0000 7fdc93fff640 10 monclient(hunting): picked mon.noname-a con 0x7fdc940acfa0 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:32.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.177+0000 7fdc93fff640 10 monclient(hunting): picked mon.noname-c con 0x7fdc94072f10 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:32.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.177+0000 7fdc93fff640 10 monclient(hunting): picked mon.noname-b con 0x7fdc940ad380 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:32.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.177+0000 7fdc93fff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc940aa9f0 con 0x7fdc940acfa0 2024-03-29T10:45:32.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.177+0000 7fdc93fff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc940ac4c0 con 0x7fdc94072f10 2024-03-29T10:45:32.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.177+0000 7fdc93fff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc940ac7a0 con 0x7fdc940ad380 2024-03-29T10:45:32.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.177+0000 7fdc93fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:32.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.177+0000 7fdc93fff640 10 monclient(hunting): authenticate will time out at 1197.418274s 2024-03-29T10:45:32.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.182+0000 7fdc92ffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fdc940acfa0 0x7fdc94141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59887/0 (socket says 172.21.15.114:59887) 2024-03-29T10:45:32.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.182+0000 7fdc92ffd640 1 -- 172.21.15.114:0/1939259678 learned_addr learned my addr 172.21.15.114:0/1939259678 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:32.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964577142 0 0) 0x7fdc940ac4c0 con 0x7fdc94072f10 2024-03-29T10:45:32.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:32.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:32.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:32.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): my global_id is 4432 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc7c003520 con 0x7fdc94072f10 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663854832 0 0) 0x7fdc940aa9f0 con 0x7fdc940acfa0 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc940ac4c0 con 0x7fdc940acfa0 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190838980 0 0) 0x7fdc940ac7a0 con 0x7fdc940ad380 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:45:32.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.183+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc940aa9f0 con 0x7fdc940ad380 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381909018 0 0) 0x7fdc7c003520 con 0x7fdc94072f10 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 >> v1:172.21.15.114:6790/0 conn(0x7fdc940ad380 legacy=0x7fdc941497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 >> v1:172.21.15.114:6789/0 conn(0x7fdc940acfa0 legacy=0x7fdc94141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 1 monclient: found mon.noname-c 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc940ab5a0 con 0x7fdc94072f10 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 10 monclient: _finish_auth 0 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdc8c001a10 con 0x7fdc94072f10 2024-03-29T10:45:32.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc91ffb640 10 monclient: dump: 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc93fff640 5 monclient: authenticate success, global_id 4432 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.187+0000 7fdc93fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:32.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.188+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc8c001b70 con 0x7fdc94072f10 2024-03-29T10:45:32.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.188+0000 7fdc91ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:32.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.188+0000 7fdc91ffb640 1 -- 172.21.15.114:0/1939259678 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdc8c001fc0 con 0x7fdc94072f10 2024-03-29T10:45:32.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.188+0000 7fdc91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:32.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.188+0000 7fdc91ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:32.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.188+0000 7fdc91ffb640 10 monclient: dump: 2024-03-29T10:45:32.174 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:32.174 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:32.174 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:32.174 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:32.174 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.188+0000 7fdc93fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.188+0000 7fdc93fff640 4 set_mon_vals no callback set 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.189+0000 7fdc93fff640 10 monclient: shutdown 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.189+0000 7fdc93fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.189+0000 7fdc93fff640 1 -- 172.21.15.114:0/1939259678 >> v1:172.21.15.176:6789/0 conn(0x7fdc94072f10 legacy=0x7fdc94141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.189+0000 7fdc93fff640 1 -- 172.21.15.114:0/1939259678 shutdown_connections 2024-03-29T10:45:32.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.189+0000 7fdc93fff640 1 -- 172.21.15.114:0/1939259678 >> 172.21.15.114:0/1939259678 conn(0x7fdc940b3a30 msgr2=0x7fdc940b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.189+0000 7fdc93fff640 1 -- 172.21.15.114:0/1939259678 shutdown_connections 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.189+0000 7fdc93fff640 1 -- 172.21.15.114:0/1939259678 wait complete. 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.190+0000 7fdc93fff640 10 monclient: build_initial_monmap 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.190+0000 7fdc93fff640 10 monclient: monmap: 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:45:32.177 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:32.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.191+0000 7fdc93fff640 1 librados: starting msgr at 2024-03-29T10:45:32.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.191+0000 7fdc93fff640 1 librados: starting objecter 2024-03-29T10:45:32.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.191+0000 7fdc93fff640 1 Processor -- start 2024-03-29T10:45:32.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.191+0000 7fdc93fff640 1 -- start start 2024-03-29T10:45:32.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.191+0000 7fdc93fff640 1 librados: setting wanted keys 2024-03-29T10:45:32.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.191+0000 7fdc93fff640 1 librados: calling monclient init 2024-03-29T10:45:32.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.191+0000 7fdc93fff640 10 monclient: init 2024-03-29T10:45:32.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.192+0000 7fdc93fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:32.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.192+0000 7fdc93fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:32.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.192+0000 7fdc93fff640 10 monclient(hunting): picked mon.noname-a con 0x7fdc94072f10 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:32.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.192+0000 7fdc93fff640 10 monclient(hunting): picked mon.noname-c con 0x7fdc940acfa0 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:32.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.192+0000 7fdc93fff640 10 monclient(hunting): picked mon.noname-b con 0x7fdc940ad380 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:32.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.192+0000 7fdc93fff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc941d3ff0 con 0x7fdc94072f10 2024-03-29T10:45:32.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.192+0000 7fdc93fff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9413f050 con 0x7fdc940ad380 2024-03-29T10:45:32.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.192+0000 7fdc93fff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc941d67b0 con 0x7fdc940acfa0 2024-03-29T10:45:32.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.192+0000 7fdc93fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:32.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.192+0000 7fdc93fff640 10 monclient(hunting): authenticate will time out at 1197.433247s 2024-03-29T10:45:32.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.198+0000 7fdc92ffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fdc94072f10 0x7fdc941d60a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34399/0 (socket says 172.21.15.114:34399) 2024-03-29T10:45:32.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.198+0000 7fdc92ffd640 1 -- 172.21.15.114:0/3276607221 learned_addr learned my addr 172.21.15.114:0/3276607221 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:32.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.198+0000 7fdc7bfff640 10 client.?.objecter ms_handle_connect 0x7fdc94072f10 2024-03-29T10:45:32.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.198+0000 7fdc7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:32.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.198+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471313255 0 0) 0x7fdc941d3ff0 con 0x7fdc94072f10 2024-03-29T10:45:32.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.198+0000 7fdc7bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:32.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.198+0000 7fdc7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:32.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.198+0000 7fdc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:32.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.198+0000 7fdc7bfff640 10 monclient(hunting): my global_id is 4461 2024-03-29T10:45:32.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.198+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc80004eb0 con 0x7fdc94072f10 2024-03-29T10:45:32.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.199+0000 7fdc7bfff640 10 client.?.objecter ms_handle_connect 0x7fdc940ad380 2024-03-29T10:45:32.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.199+0000 7fdc7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:32.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.199+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28715851 0 0) 0x7fdc9413f050 con 0x7fdc940ad380 2024-03-29T10:45:32.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.199+0000 7fdc7bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:32.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.199+0000 7fdc7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:32.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.199+0000 7fdc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:32.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.199+0000 7fdc7bfff640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:45:32.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.199+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc941d3ff0 con 0x7fdc940ad380 2024-03-29T10:45:32.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 client.?.objecter ms_handle_connect 0x7fdc940acfa0 2024-03-29T10:45:32.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:32.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4054008939 0 0) 0x7fdc80004eb0 con 0x7fdc94072f10 2024-03-29T10:45:32.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc9413f050 con 0x7fdc94072f10 2024-03-29T10:45:32.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdc84001980 con 0x7fdc94072f10 2024-03-29T10:45:32.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient(hunting): dump: 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:32.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864578866 0 0) 0x7fdc941d67b0 con 0x7fdc940acfa0 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc80004eb0 con 0x7fdc940acfa0 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2919335869 0 0) 0x7fdc9413f050 con 0x7fdc94072f10 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 >> v1:172.21.15.114:6790/0 conn(0x7fdc940acfa0 legacy=0x7fdc94135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 >> v1:172.21.15.176:6789/0 conn(0x7fdc940ad380 legacy=0x7fdc941ff970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 1 monclient: found mon.a 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc941d6940 con 0x7fdc94072f10 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient: _finish_auth 0 2024-03-29T10:45:32.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc93fff640 5 monclient: authenticate success, global_id 4461 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc93fff640 10 monclient: _renew_subs 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc93fff640 1 -- 172.21.15.114:0/3276607221 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc941d3a40 con 0x7fdc94072f10 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc93fff640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc93fff640 10 monclient: _renew_subs 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc93fff640 1 -- 172.21.15.114:0/3276607221 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc94135c60 con 0x7fdc94072f10 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.201+0000 7fdc93fff640 1 librados: init done 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.202+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc840047e0 con 0x7fdc94072f10 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.202+0000 7fdc7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.202+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdc84004af0 con 0x7fdc94072f10 2024-03-29T10:45:32.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.202+0000 7fdc7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.202+0000 7fdc7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.202+0000 7fdc90ff9640 4 set_mon_vals no callback set 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.202+0000 7fdc7bfff640 10 monclient: dump: 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:32.189 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:32.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.203+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fdc8401aac0 con 0x7fdc94072f10 2024-03-29T10:45:32.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.204+0000 7fdc927fc640 10 monclient: get_auth_request con 0x7fdc8006de80 auth_method 0 2024-03-29T10:45:32.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.204+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fdc84084a90 con 0x7fdc94072f10 2024-03-29T10:45:32.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.204+0000 7fdc7bfff640 10 client.4461.objecter ms_dispatch 0x7fdc94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:32.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.204+0000 7fdc7bfff640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:32.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.204+0000 7fdc7bfff640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:32.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.204+0000 7fdc7bfff640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T10:45:32.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.205+0000 7fdc93fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:32.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.205+0000 7fdc93fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:32.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.205+0000 7fdc93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:45:32.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.205+0000 7fdc93fff640 1 -- 172.21.15.114:0/3276607221 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc74005080 con 0x7fdc94072f10 2024-03-29T10:45:32.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.207+0000 7fdc7bfff640 10 client.4461.objecter ms_handle_connect 0x7fdc8006de80 2024-03-29T10:45:32.198 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.215+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc840501a0 con 0x7fdc94072f10 2024-03-29T10:45:32.198 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.215+0000 7fdc7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:32.198 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.215+0000 7fdc7bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:32.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.533+0000 7fdc93fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:32.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.533+0000 7fdc93fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:32.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.533+0000 7fdc93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:45:32.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.533+0000 7fdc93fff640 1 -- 172.21.15.114:0/3276607221 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc74005330 con 0x7fdc94072f10 2024-03-29T10:45:32.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.534+0000 7fdc7bfff640 1 -- 172.21.15.114:0/3276607221 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fdc840522a0 con 0x7fdc94072f10 2024-03-29T10:45:32.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.534+0000 7fdc7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:32.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.534+0000 7fdc7bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:32.519 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:45:32.519 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:32.521 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.538+0000 7fdc93fff640 10 librados: watch_flush enter 2024-03-29T10:45:32.523 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.538+0000 7fdc93fff640 10 librados: watch_flush exit 2024-03-29T10:45:32.523 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.538+0000 7fdc93fff640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T10:45:32.523 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.538+0000 7fdc93fff640 10 client.4461.objecter successfully canceled tick 2024-03-29T10:45:32.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.541+0000 7fdc93fff640 1 -- 172.21.15.114:0/3276607221 >> v1:172.21.15.114:6816/775044307 conn(0x7fdc8006de80 legacy=0x7fdc80070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:32.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.541+0000 7fdc93fff640 10 monclient: shutdown 2024-03-29T10:45:32.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.541+0000 7fdc93fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:32.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.541+0000 7fdc93fff640 1 -- 172.21.15.114:0/3276607221 >> v1:172.21.15.114:6789/0 conn(0x7fdc94072f10 legacy=0x7fdc941d60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:32.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.541+0000 7fdc93fff640 1 -- 172.21.15.114:0/3276607221 shutdown_connections 2024-03-29T10:45:32.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.541+0000 7fdc93fff640 1 -- 172.21.15.114:0/3276607221 >> 172.21.15.114:0/3276607221 conn(0x7fdc940b3a30 msgr2=0x7fdc940b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:32.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.545+0000 7fdc93fff640 1 -- 172.21.15.114:0/3276607221 shutdown_connections 2024-03-29T10:45:32.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.548+0000 7fdc93fff640 1 -- 172.21.15.114:0/3276607221 wait complete. 2024-03-29T10:45:32.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:32.549+0000 7fdc93fff640 1 librados: shutdown 2024-03-29T10:45:32.552 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:39.360 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:39.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.534+0000 7f8d1bfff640 10 monclient: get_monmap_and_config 2024-03-29T10:45:39.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.534+0000 7f8d1bfff640 10 monclient: build_initial_monmap 2024-03-29T10:45:39.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.534+0000 7f8d1bfff640 10 monclient: monmap: 2024-03-29T10:45:39.518 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:39.518 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:39.519 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:45:39.536206+0000 2024-03-29T10:45:39.519 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:45:39.536206+0000 2024-03-29T10:45:39.519 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:39.519 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:39.519 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:39.519 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:45:39.519 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:45:39.519 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:39.521 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.539+0000 7f8d1bfff640 1 Processor -- start 2024-03-29T10:45:39.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.540+0000 7f8d1bfff640 1 -- start start 2024-03-29T10:45:39.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.540+0000 7f8d1bfff640 10 monclient: init 2024-03-29T10:45:39.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.541+0000 7f8d1bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:39.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.541+0000 7f8d1bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:39.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.541+0000 7f8d1bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f8d1c0aac30 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:39.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.541+0000 7f8d1bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f8d1c072f20 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:39.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.541+0000 7f8d1bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f8d1c0a9e90 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:39.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.541+0000 7f8d1bfff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c0ad8e0 con 0x7f8d1c0aac30 2024-03-29T10:45:39.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.541+0000 7f8d1bfff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c0ac6a0 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.541+0000 7f8d1bfff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c0ab230 con 0x7f8d1c072f20 2024-03-29T10:45:39.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.541+0000 7f8d1bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:39.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.541+0000 7f8d1bfff640 10 monclient(hunting): authenticate will time out at 1204.782632s 2024-03-29T10:45:39.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.542+0000 7f8d1affd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8d1c0aac30 0x7f8d1c13af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49172/0 (socket says 172.21.15.114:49172) 2024-03-29T10:45:39.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.543+0000 7f8d1affd640 1 -- 172.21.15.114:0/2736333695 learned_addr learned my addr 172.21.15.114:0/2736333695 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:39.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.544+0000 7f8d19ffb640 1 -- 172.21.15.114:0/2736333695 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267788371 0 0) 0x7f8d1c0ad8e0 con 0x7f8d1c0aac30 2024-03-29T10:45:39.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.544+0000 7f8d19ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:39.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.544+0000 7f8d19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:39.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.544+0000 7f8d19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:39.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.544+0000 7f8d19ffb640 10 monclient(hunting): my global_id is 4464 2024-03-29T10:45:39.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.544+0000 7f8d19ffb640 1 -- 172.21.15.114:0/2736333695 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d04003520 con 0x7f8d1c0aac30 2024-03-29T10:45:39.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 1 -- 172.21.15.114:0/2736333695 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2556860604 0 0) 0x7f8d04003520 con 0x7f8d1c0aac30 2024-03-29T10:45:39.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 1 -- 172.21.15.114:0/2736333695 >> v1:172.21.15.114:6790/0 conn(0x7f8d1c072f20 legacy=0x7f8d1c0acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:39.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 1 -- 172.21.15.114:0/2736333695 >> v1:172.21.15.176:6789/0 conn(0x7f8d1c0a9e90 legacy=0x7f8d1c0ad1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:39.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:45:39.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 1 monclient: found mon.noname-a 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 1 -- 172.21.15.114:0/2736333695 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c13b660 con 0x7f8d1c0aac30 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 10 monclient: _finish_auth 0 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 1 -- 172.21.15.114:0/2736333695 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8d14001a10 con 0x7f8d1c0aac30 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.545+0000 7f8d19ffb640 10 monclient: dump: 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:39.529 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:39.530 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:39.530 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:39.530 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:39.530 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:39.530 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:39.530 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:39.530 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:39.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.546+0000 7f8d1bfff640 5 monclient: authenticate success, global_id 4464 2024-03-29T10:45:39.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.546+0000 7f8d1bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:39.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.546+0000 7f8d19ffb640 1 -- 172.21.15.114:0/2736333695 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d14001ca0 con 0x7f8d1c0aac30 2024-03-29T10:45:39.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.546+0000 7f8d19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:39.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.546+0000 7f8d19ffb640 1 -- 172.21.15.114:0/2736333695 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8d14002030 con 0x7f8d1c0aac30 2024-03-29T10:45:39.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.546+0000 7f8d1bfff640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:39.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.546+0000 7f8d1bfff640 4 set_mon_vals no callback set 2024-03-29T10:45:39.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.547+0000 7f8d1bfff640 10 monclient: shutdown 2024-03-29T10:45:39.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.547+0000 7f8d1bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:39.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.547+0000 7f8d1bfff640 1 -- 172.21.15.114:0/2736333695 >> v1:172.21.15.114:6789/0 conn(0x7f8d1c0aac30 legacy=0x7f8d1c13af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:39.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.547+0000 7f8d19ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:45:39.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.547+0000 7f8d1bfff640 1 -- 172.21.15.114:0/2736333695 shutdown_connections 2024-03-29T10:45:39.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.547+0000 7f8d1bfff640 1 -- 172.21.15.114:0/2736333695 >> 172.21.15.114:0/2736333695 conn(0x7f8d1c0b3a30 msgr2=0x7f8d1c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.548+0000 7f8d1bfff640 1 -- 172.21.15.114:0/2736333695 shutdown_connections 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.548+0000 7f8d1bfff640 1 -- 172.21.15.114:0/2736333695 wait complete. 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.549+0000 7f8d1bfff640 10 monclient: build_initial_monmap 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.549+0000 7f8d1bfff640 10 monclient: monmap: 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:45:39.533 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.550+0000 7f8d1bfff640 1 librados: starting msgr at 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.550+0000 7f8d1bfff640 1 librados: starting objecter 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.550+0000 7f8d1bfff640 1 Processor -- start 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.550+0000 7f8d1bfff640 1 -- start start 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.550+0000 7f8d1bfff640 1 librados: setting wanted keys 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.550+0000 7f8d1bfff640 1 librados: calling monclient init 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.550+0000 7f8d1bfff640 10 monclient: init 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.551+0000 7f8d1bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.551+0000 7f8d1bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.552+0000 7f8d1bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f8d1c072f20 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.552+0000 7f8d1affd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8d1c072f20 0x7f8d1c1497c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51530/0 (socket says 172.21.15.114:51530) 2024-03-29T10:45:39.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.552+0000 7f8d1affd640 1 -- 172.21.15.114:0/1476497143 learned_addr learned my addr 172.21.15.114:0/1476497143 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:39.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.552+0000 7f8d1bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f8d1c0a9e90 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:39.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.552+0000 7f8d03fff640 10 client.?.objecter ms_handle_connect 0x7f8d1c072f20 2024-03-29T10:45:39.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.556+0000 7f8d03fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:39.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d1bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f8d1c0aac30 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:39.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c12f4b0 con 0x7f8d1c072f20 2024-03-29T10:45:39.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c12d6a0 con 0x7f8d1c0aac30 2024-03-29T10:45:39.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c12d910 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d1bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:39.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d1bfff640 10 monclient(hunting): authenticate will time out at 1204.799117s 2024-03-29T10:45:39.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d03fff640 10 client.?.objecter ms_handle_connect 0x7f8d1c0a9e90 2024-03-29T10:45:39.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d03fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:39.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683266390 0 0) 0x7f8d1c12d910 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d03fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:39.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:39.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:39.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d03fff640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:45:39.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.558+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d08004eb0 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.559+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1211560717 0 0) 0x7f8d08004eb0 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.559+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d1c12d910 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.559+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8d10001870 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.559+0000 7f8d03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:39.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.559+0000 7f8d03fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:39.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.559+0000 7f8d03fff640 10 monclient(hunting): dump: 2024-03-29T10:45:39.542 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:39.543 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:39.543 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:39.543 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:39.543 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:39.543 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:39.543 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:39.543 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:39.543 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:39.543 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:39.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.559+0000 7f8d03fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.560+0000 7f8d03fff640 10 client.?.objecter ms_handle_connect 0x7f8d1c0aac30 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.560+0000 7f8d03fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.560+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713229847 0 0) 0x7f8d1c12f4b0 con 0x7f8d1c072f20 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.560+0000 7f8d03fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.560+0000 7f8d03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.560+0000 7f8d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.560+0000 7f8d03fff640 10 monclient(hunting): my global_id is 4467 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.560+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d08005680 con 0x7f8d1c072f20 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.560+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 60663713 0 0) 0x7f8d1c12d910 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.560+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 >> v1:172.21.15.176:6789/0 conn(0x7f8d1c0aac30 legacy=0x7f8d1c1e6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.560+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 >> v1:172.21.15.114:6789/0 conn(0x7f8d1c072f20 legacy=0x7f8d1c1497c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:39.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d03fff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d03fff640 1 monclient: found mon.c 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c12dbf0 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d03fff640 10 monclient: _finish_auth 0 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d03fff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d1bfff640 5 monclient: authenticate success, global_id 4511 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d1bfff640 10 monclient: _renew_subs 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d1bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1c12f760 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d1bfff640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d1bfff640 10 monclient: _renew_subs 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d1bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:39.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.561+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1c148e90 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.562+0000 7f8d1bfff640 1 librados: init done 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.562+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d100035f0 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.562+0000 7f8d03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.562+0000 7f8d197fa640 4 set_mon_vals no callback set 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.563+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8d10001870 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.563+0000 7f8d03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.563+0000 7f8d03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.563+0000 7f8d03fff640 10 monclient: dump: 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:39.549 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:39.550 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:39.550 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:39.550 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:39.550 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:39.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.564+0000 7f8d1bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:39.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.566+0000 7f8d1bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:39.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.566+0000 7f8d1bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:39.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.566+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cfc005080 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.567+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f8d100195d0 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.568+0000 7f8d1affd640 10 monclient: get_auth_request con 0x7f8d0806e320 auth_method 0 2024-03-29T10:45:39.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.568+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f8d10053190 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.568+0000 7f8d03fff640 10 client.4511.objecter ms_dispatch 0x7f8d1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:39.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.568+0000 7f8d03fff640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:39.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.568+0000 7f8d03fff640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:39.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.568+0000 7f8d03fff640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T10:45:39.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.568+0000 7f8d03fff640 10 client.4511.objecter ms_handle_connect 0x7f8d0806e320 2024-03-29T10:45:39.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.580+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d1004f430 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.580+0000 7f8d03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:39.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.580+0000 7f8d03fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:39.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.915+0000 7f8d1bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:39.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.915+0000 7f8d1bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:39.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.915+0000 7f8d1bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:45:39.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.915+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cfc005330 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.917+0000 7f8d03fff640 1 -- 172.21.15.114:0/1476497143 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f8d1004f430 con 0x7f8d1c0a9e90 2024-03-29T10:45:39.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.917+0000 7f8d03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:39.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.917+0000 7f8d03fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:39.900 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:45:39.900 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:39.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.920+0000 7f8d1bfff640 10 librados: watch_flush enter 2024-03-29T10:45:39.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.921+0000 7f8d1bfff640 10 librados: watch_flush exit 2024-03-29T10:45:39.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.921+0000 7f8d1bfff640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T10:45:39.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.921+0000 7f8d1bfff640 10 client.4511.objecter successfully canceled tick 2024-03-29T10:45:39.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.921+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 >> v1:172.21.15.114:6816/775044307 conn(0x7f8d0806e320 legacy=0x7f8d080707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:39.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.921+0000 7f8d1bfff640 10 monclient: shutdown 2024-03-29T10:45:39.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.921+0000 7f8d1bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:39.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.921+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 >> v1:172.21.15.114:6790/0 conn(0x7f8d1c0a9e90 legacy=0x7f8d1c12e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:39.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.921+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 shutdown_connections 2024-03-29T10:45:39.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.921+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 >> 172.21.15.114:0/1476497143 conn(0x7f8d1c0b3a30 msgr2=0x7f8d1c0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:39.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.921+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 shutdown_connections 2024-03-29T10:45:39.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.922+0000 7f8d1bfff640 1 -- 172.21.15.114:0/1476497143 wait complete. 2024-03-29T10:45:39.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:39.922+0000 7f8d1bfff640 1 librados: shutdown 2024-03-29T10:45:39.924 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:46.731 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:46.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.898+0000 7fa08ca0e640 10 monclient: get_monmap_and_config 2024-03-29T10:45:46.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.898+0000 7fa08ca0e640 10 monclient: build_initial_monmap 2024-03-29T10:45:46.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.898+0000 7fa08ca0e640 10 monclient: monmap: 2024-03-29T10:45:46.882 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:46.882 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:46.882 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:45:46.899672+0000 2024-03-29T10:45:46.882 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:45:46.899672+0000 2024-03-29T10:45:46.882 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:46.882 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:46.882 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:46.882 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:45:46.882 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:45:46.882 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:46.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.900+0000 7fa08ca0e640 1 Processor -- start 2024-03-29T10:45:46.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.900+0000 7fa08ca0e640 1 -- start start 2024-03-29T10:45:46.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.900+0000 7fa08ca0e640 10 monclient: init 2024-03-29T10:45:46.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.901+0000 7fa08ca0e640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:46.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.901+0000 7fa08ca0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:46.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.901+0000 7fa08ca0e640 10 monclient(hunting): picked mon.noname-b con 0x7fa0880a9750 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:46.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.902+0000 7fa08ca0e640 10 monclient(hunting): picked mon.noname-c con 0x7fa0880af2e0 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:46.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.902+0000 7fa08ca0e640 10 monclient(hunting): picked mon.noname-a con 0x7fa088141ef0 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:46.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.902+0000 7fa08ca0e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0880a9b30 con 0x7fa088141ef0 2024-03-29T10:45:46.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.902+0000 7fa08ca0e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0880ac4f0 con 0x7fa0880af2e0 2024-03-29T10:45:46.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.902+0000 7fa08ca0e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0880af770 con 0x7fa0880a9750 2024-03-29T10:45:46.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.902+0000 7fa08ca0e640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:46.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.902+0000 7fa08ca0e640 10 monclient(hunting): authenticate will time out at 1212.143975s 2024-03-29T10:45:46.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.903+0000 7fa086d9d640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa088141ef0 0x7fa0881498e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55060/0 (socket says 172.21.15.114:55060) 2024-03-29T10:45:46.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.903+0000 7fa086d9d640 1 -- 172.21.15.114:0/3366043784 learned_addr learned my addr 172.21.15.114:0/3366043784 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:46.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 1 -- 172.21.15.114:0/3366043784 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812695451 0 0) 0x7fa0880a9b30 con 0x7fa088141ef0 2024-03-29T10:45:46.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:46.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:46.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:46.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 10 monclient(hunting): my global_id is 4470 2024-03-29T10:45:46.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 1 -- 172.21.15.114:0/3366043784 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa070003520 con 0x7fa088141ef0 2024-03-29T10:45:46.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 1 -- 172.21.15.114:0/3366043784 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255352890 0 0) 0x7fa0880af770 con 0x7fa0880a9750 2024-03-29T10:45:46.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:46.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:46.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:46.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 10 monclient(hunting): my global_id is 4514 2024-03-29T10:45:46.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 1 -- 172.21.15.114:0/3366043784 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0880a9b30 con 0x7fa0880a9750 2024-03-29T10:45:46.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 1 -- 172.21.15.114:0/3366043784 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2020343042 0 0) 0x7fa070003520 con 0x7fa088141ef0 2024-03-29T10:45:46.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 1 -- 172.21.15.114:0/3366043784 >> v1:172.21.15.114:6790/0 conn(0x7fa0880a9750 legacy=0x7fa08813b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:46.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 1 -- 172.21.15.114:0/3366043784 >> v1:172.21.15.176:6789/0 conn(0x7fa0880af2e0 legacy=0x7fa0881417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:46.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 10 monclient: _finish_hunting 0 2024-03-29T10:45:46.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 1 monclient: found mon.noname-a 2024-03-29T10:45:46.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:46.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:45:46.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.906+0000 7fa08559a640 1 -- 172.21.15.114:0/3366043784 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0880b00c0 con 0x7fa088141ef0 2024-03-29T10:45:46.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 10 monclient: _finish_auth 0 2024-03-29T10:45:46.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 10 monclient: _check_auth_tickets 2024-03-29T10:45:46.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:46.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 1 -- 172.21.15.114:0/3366043784 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa078001690 con 0x7fa088141ef0 2024-03-29T10:45:46.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:46.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:46.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 10 monclient: dump: 2024-03-29T10:45:46.894 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:46.894 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:46.894 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:46.894 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:46.894 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:46.895 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:46.895 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:46.895 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:46.895 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:46.895 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:46.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08ca0e640 5 monclient: authenticate success, global_id 4470 2024-03-29T10:45:46.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08ca0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:46.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 1 -- 172.21.15.114:0/3366043784 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0780018b0 con 0x7fa088141ef0 2024-03-29T10:45:46.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:46.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 1 -- 172.21.15.114:0/3366043784 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa078001bc0 con 0x7fa088141ef0 2024-03-29T10:45:46.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08ca0e640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:46.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08ca0e640 4 set_mon_vals no callback set 2024-03-29T10:45:46.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:46.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:46.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.907+0000 7fa08559a640 10 monclient: dump: 2024-03-29T10:45:46.912 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:46.912 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:46.913 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:46.913 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:46.913 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:46.913 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:46.913 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:46.913 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:46.914 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:46.914 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:46.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.908+0000 7fa08ca0e640 10 monclient: shutdown 2024-03-29T10:45:46.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.908+0000 7fa08ca0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:46.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.908+0000 7fa08ca0e640 1 -- 172.21.15.114:0/3366043784 >> v1:172.21.15.114:6789/0 conn(0x7fa088141ef0 legacy=0x7fa0881498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:46.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.908+0000 7fa08ca0e640 1 -- 172.21.15.114:0/3366043784 shutdown_connections 2024-03-29T10:45:46.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.908+0000 7fa08ca0e640 1 -- 172.21.15.114:0/3366043784 >> 172.21.15.114:0/3366043784 conn(0x7fa0880b3a00 msgr2=0x7fa0880b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:46.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.909+0000 7fa08ca0e640 1 -- 172.21.15.114:0/3366043784 shutdown_connections 2024-03-29T10:45:46.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.909+0000 7fa08ca0e640 1 -- 172.21.15.114:0/3366043784 wait complete. 2024-03-29T10:45:46.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.911+0000 7fa08ca0e640 10 monclient: build_initial_monmap 2024-03-29T10:45:46.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.912+0000 7fa08ca0e640 10 monclient: monmap: 2024-03-29T10:45:46.916 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:46.916 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:46.916 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:45:46.916 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:45:46.916 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:46.916 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.912+0000 7fa08ca0e640 1 librados: starting msgr at 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.912+0000 7fa08ca0e640 1 librados: starting objecter 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.912+0000 7fa08ca0e640 1 Processor -- start 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.913+0000 7fa08ca0e640 1 -- start start 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.913+0000 7fa08ca0e640 1 librados: setting wanted keys 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.913+0000 7fa08ca0e640 1 librados: calling monclient init 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.913+0000 7fa08ca0e640 10 monclient: init 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.913+0000 7fa08ca0e640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.914+0000 7fa08ca0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:46.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.917+0000 7fa08ca0e640 10 monclient(hunting): picked mon.noname-b con 0x7fa0880a9750 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.917+0000 7fa08ca0e640 10 monclient(hunting): picked mon.noname-c con 0x7fa088141ef0 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.917+0000 7fa08ca0e640 10 monclient(hunting): picked mon.noname-a con 0x7fa08813da10 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.917+0000 7fa08ca0e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08806e440 con 0x7fa08813da10 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.917+0000 7fa08ca0e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08813e880 con 0x7fa0880a9750 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.917+0000 7fa08ca0e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa088136ce0 con 0x7fa088141ef0 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.917+0000 7fa08ca0e640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.918+0000 7fa08ca0e640 10 monclient(hunting): authenticate will time out at 1212.159046s 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.918+0000 7fa08659c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fa0880a9750 0x7fa08813e030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:38785/0 (socket says 172.21.15.114:38785) 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.918+0000 7fa08659c640 1 -- 172.21.15.114:0/4052443921 learned_addr learned my addr 172.21.15.114:0/4052443921 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.918+0000 7fa076ffd640 10 client.?.objecter ms_handle_connect 0x7fa0880a9750 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.918+0000 7fa076ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.918+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130669723 0 0) 0x7fa08813e880 con 0x7fa0880a9750 2024-03-29T10:45:46.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.918+0000 7fa076ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.918+0000 7fa076ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.918+0000 7fa076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.918+0000 7fa076ffd640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.918+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa068004eb0 con 0x7fa0880a9750 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.919+0000 7fa076ffd640 10 client.?.objecter ms_handle_connect 0x7fa08813da10 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.919+0000 7fa076ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.919+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2944801448 0 0) 0x7fa068004eb0 con 0x7fa0880a9750 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.919+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa08813e880 con 0x7fa0880a9750 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.919+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa07c001a10 con 0x7fa0880a9750 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.919+0000 7fa076ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.919+0000 7fa076ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.919+0000 7fa076ffd640 10 monclient(hunting): dump: 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:46.919 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.919+0000 7fa076ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.920+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1908037306 0 0) 0x7fa08813e880 con 0x7fa0880a9750 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.920+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 >> v1:172.21.15.114:6790/0 conn(0x7fa088141ef0 legacy=0x7fa0881338b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.920+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 >> v1:172.21.15.114:6789/0 conn(0x7fa08813da10 legacy=0x7fa0881d2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.920+0000 7fa076ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:46.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.920+0000 7fa076ffd640 1 monclient: found mon.b 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.920+0000 7fa076ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.920+0000 7fa076ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.920+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa088136e70 con 0x7fa0880a9750 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.920+0000 7fa076ffd640 10 monclient: _finish_auth 0 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.920+0000 7fa076ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.920+0000 7fa076ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.921+0000 7fa08ca0e640 5 monclient: authenticate success, global_id 4444 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.921+0000 7fa08ca0e640 10 monclient: _renew_subs 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.921+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa07c001b70 con 0x7fa0880a9750 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.921+0000 7fa08ca0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.921+0000 7fa08ca0e640 1 -- 172.21.15.114:0/4052443921 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa088133fc0 con 0x7fa0880a9750 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.921+0000 7fa08ca0e640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:46.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.921+0000 7fa076ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.921+0000 7fa084d99640 4 set_mon_vals no callback set 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.922+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fa07c003a60 con 0x7fa0880a9750 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.922+0000 7fa076ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.923+0000 7fa076ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.923+0000 7fa076ffd640 10 monclient: dump: 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:46.922 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.923+0000 7fa08ca0e640 10 monclient: _renew_subs 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.923+0000 7fa08ca0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.923+0000 7fa08ca0e640 1 -- 172.21.15.114:0/4052443921 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0881d3690 con 0x7fa0880a9750 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.923+0000 7fa08ca0e640 1 librados: init done 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.923+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fa07c003c80 con 0x7fa0880a9750 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.924+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fa07c085c60 con 0x7fa0880a9750 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.924+0000 7fa076ffd640 10 client.4444.objecter ms_dispatch 0x7fa088000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.924+0000 7fa076ffd640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.924+0000 7fa076ffd640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.924+0000 7fa076ffd640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.924+0000 7fa08ca0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:46.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.924+0000 7fa085d9b640 10 monclient: get_auth_request con 0x7fa06806dc30 auth_method 0 2024-03-29T10:45:46.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.924+0000 7fa08ca0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:46.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.924+0000 7fa08ca0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:45:46.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.925+0000 7fa08ca0e640 1 -- 172.21.15.114:0/4052443921 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa088134150 con 0x7fa0880a9750 2024-03-29T10:45:46.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.925+0000 7fa076ffd640 10 client.4444.objecter ms_handle_connect 0x7fa06806dc30 2024-03-29T10:45:46.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.931+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa07c050510 con 0x7fa0880a9750 2024-03-29T10:45:46.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.931+0000 7fa076ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:46.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:46.931+0000 7fa076ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:47.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.268+0000 7fa08ca0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:47.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.268+0000 7fa08ca0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:47.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.268+0000 7fa08ca0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:45:47.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.268+0000 7fa08ca0e640 1 -- 172.21.15.114:0/4052443921 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa08804f730 con 0x7fa0880a9750 2024-03-29T10:45:47.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.269+0000 7fa076ffd640 1 -- 172.21.15.114:0/4052443921 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fa07c052440 con 0x7fa0880a9750 2024-03-29T10:45:47.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.269+0000 7fa076ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:47.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.269+0000 7fa076ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:47.253 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:45:47.253 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:47.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.273+0000 7fa08ca0e640 10 librados: watch_flush enter 2024-03-29T10:45:47.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 10 librados: watch_flush exit 2024-03-29T10:45:47.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-29T10:45:47.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 10 client.4444.objecter successfully canceled tick 2024-03-29T10:45:47.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 1 -- 172.21.15.114:0/4052443921 >> v1:172.21.15.114:6816/775044307 conn(0x7fa06806dc30 legacy=0x7fa0680700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:47.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 10 monclient: shutdown 2024-03-29T10:45:47.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:47.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 1 -- 172.21.15.114:0/4052443921 >> v1:172.21.15.176:6789/0 conn(0x7fa0880a9750 legacy=0x7fa08813e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:47.260 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 1 -- 172.21.15.114:0/4052443921 shutdown_connections 2024-03-29T10:45:47.260 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 1 -- 172.21.15.114:0/4052443921 >> 172.21.15.114:0/4052443921 conn(0x7fa0880b3a00 msgr2=0x7fa088069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:47.260 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 1 -- 172.21.15.114:0/4052443921 shutdown_connections 2024-03-29T10:45:47.260 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 1 -- 172.21.15.114:0/4052443921 wait complete. 2024-03-29T10:45:47.260 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:47.274+0000 7fa08ca0e640 1 librados: shutdown 2024-03-29T10:45:47.276 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:54.083 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.263+0000 7fb46457b640 10 monclient: get_monmap_and_config 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.263+0000 7fb46457b640 10 monclient: build_initial_monmap 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.263+0000 7fb46457b640 10 monclient: monmap: 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:45:54.263302+0000 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:45:54.263302+0000 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:45:54.245 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:45:54.246 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:54.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.264+0000 7fb46457b640 1 Processor -- start 2024-03-29T10:45:54.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.264+0000 7fb46457b640 1 -- start start 2024-03-29T10:45:54.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.264+0000 7fb46457b640 10 monclient: init 2024-03-29T10:45:54.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.265+0000 7fb46457b640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:54.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.265+0000 7fb46457b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:54.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.265+0000 7fb46457b640 10 monclient(hunting): picked mon.noname-c con 0x7fb45c0b0860 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:54.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.265+0000 7fb46457b640 10 monclient(hunting): picked mon.noname-a con 0x7fb45c13ef30 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:54.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.265+0000 7fb46457b640 10 monclient(hunting): picked mon.noname-b con 0x7fb45c0af8e0 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:54.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.266+0000 7fb46457b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c0b0c40 con 0x7fb45c13ef30 2024-03-29T10:45:54.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.266+0000 7fb46457b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c0aed10 con 0x7fb45c0b0860 2024-03-29T10:45:54.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.266+0000 7fb46457b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c0afd50 con 0x7fb45c0af8e0 2024-03-29T10:45:54.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.266+0000 7fb46457b640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:54.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.266+0000 7fb46457b640 10 monclient(hunting): authenticate will time out at 1219.506165s 2024-03-29T10:45:54.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.266+0000 7fb461b16640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb45c13ef30 0x7fb45c0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48260/0 (socket says 172.21.15.114:48260) 2024-03-29T10:45:54.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.266+0000 7fb461b16640 1 -- 172.21.15.114:0/1109710179 learned_addr learned my addr 172.21.15.114:0/1109710179 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:54.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292895047 0 0) 0x7fb45c0aed10 con 0x7fb45c0b0860 2024-03-29T10:45:54.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): _negotiate 2024-03-29T10:45:54.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb448003520 con 0x7fb45c0b0860 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741214788 0 0) 0x7fb45c0b0c40 con 0x7fb45c13ef30 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): _negotiate 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): my global_id is 4476 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb45c0aed10 con 0x7fb45c13ef30 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711254671 0 0) 0x7fb45c0afd50 con 0x7fb45c0af8e0 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): _negotiate 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:54.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 10 monclient(hunting): my global_id is 4517 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb45c0b0c40 con 0x7fb45c0af8e0 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.267+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4260198575 0 0) 0x7fb448003520 con 0x7fb45c0b0860 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 >> v1:172.21.15.114:6790/0 conn(0x7fb45c0af8e0 legacy=0x7fb45c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 >> v1:172.21.15.114:6789/0 conn(0x7fb45c13ef30 legacy=0x7fb45c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 10 monclient: _finish_hunting 0 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 1 monclient: found mon.noname-c 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c0ab1b0 con 0x7fb45c0b0860 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 10 monclient: _finish_auth 0 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 10 monclient: _check_auth_tickets 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb458001690 con 0x7fb45c0b0860 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb46457b640 5 monclient: authenticate success, global_id 4447 2024-03-29T10:45:54.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb46457b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb461315640 10 monclient: dump: 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:54.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.268+0000 7fb46457b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.269+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4580018b0 con 0x7fb45c0b0860 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.269+0000 7fb461315640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.269+0000 7fb461315640 1 -- 172.21.15.114:0/1109710179 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb458001c40 con 0x7fb45c0b0860 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.269+0000 7fb46457b640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.269+0000 7fb46457b640 4 set_mon_vals no callback set 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.270+0000 7fb46457b640 10 monclient: shutdown 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.270+0000 7fb461315640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.270+0000 7fb461315640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.270+0000 7fb461315640 10 monclient: dump: 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.256 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.270+0000 7fb46457b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.270+0000 7fb46457b640 1 -- 172.21.15.114:0/1109710179 >> v1:172.21.15.176:6789/0 conn(0x7fb45c0b0860 legacy=0x7fb45c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.270+0000 7fb46457b640 1 -- 172.21.15.114:0/1109710179 shutdown_connections 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.270+0000 7fb46457b640 1 -- 172.21.15.114:0/1109710179 >> 172.21.15.114:0/1109710179 conn(0x7fb45c06b660 msgr2=0x7fb45c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.270+0000 7fb46457b640 1 -- 172.21.15.114:0/1109710179 shutdown_connections 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.271+0000 7fb46457b640 1 -- 172.21.15.114:0/1109710179 wait complete. 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.272+0000 7fb46457b640 10 monclient: build_initial_monmap 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.272+0000 7fb46457b640 10 monclient: monmap: 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:45:54.257 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.273+0000 7fb46457b640 1 librados: starting msgr at 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.273+0000 7fb46457b640 1 librados: starting objecter 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.274+0000 7fb46457b640 1 Processor -- start 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.274+0000 7fb46457b640 1 -- start start 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.274+0000 7fb46457b640 1 librados: setting wanted keys 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.274+0000 7fb46457b640 1 librados: calling monclient init 2024-03-29T10:45:54.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.274+0000 7fb46457b640 10 monclient: init 2024-03-29T10:45:54.266 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.276+0000 7fb46457b640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:54.266 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.276+0000 7fb46457b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:54.266 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.276+0000 7fb46457b640 10 monclient(hunting): picked mon.noname-c con 0x7fb45c0af8e0 addr v1:172.21.15.114:6790/0 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.276+0000 7fb462317640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fb45c0af8e0 0x7fb45c0b3e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:40702/0 (socket says 172.21.15.114:40702) 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.276+0000 7fb462317640 1 -- 172.21.15.114:0/1793561930 learned_addr learned my addr 172.21.15.114:0/1793561930 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.276+0000 7fb46457b640 10 monclient(hunting): picked mon.noname-a con 0x7fb45c0b0860 addr v1:172.21.15.114:6789/0 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.276+0000 7fb452ffd640 10 client.?.objecter ms_handle_connect 0x7fb45c0af8e0 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.276+0000 7fb452ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.276+0000 7fb46457b640 10 monclient(hunting): picked mon.noname-b con 0x7fb45c13ef30 addr v1:172.21.15.176:6789/0 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.276+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c1d64a0 con 0x7fb45c0b0860 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.276+0000 7fb452ffd640 10 client.?.objecter ms_handle_connect 0x7fb45c13ef30 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c0b11d0 con 0x7fb45c13ef30 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c0b1360 con 0x7fb45c0af8e0 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 client.?.objecter ms_handle_connect 0x7fb45c0b0860 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:54.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb46457b640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654925655 0 0) 0x7fb45c0b11d0 con 0x7fb45c13ef30 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb46457b640 10 monclient(hunting): authenticate will time out at 1219.517314s 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb440004eb0 con 0x7fb45c13ef30 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707748409 0 0) 0x7fb45c1d64a0 con 0x7fb45c0b0860 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb45c0b11d0 con 0x7fb45c0b0860 2024-03-29T10:45:54.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794634461 0 0) 0x7fb45c0b1360 con 0x7fb45c0af8e0 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 10 monclient(hunting): my global_id is 4520 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.277+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb45c1d64a0 con 0x7fb45c0af8e0 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1974894584 0 0) 0x7fb440004eb0 con 0x7fb45c13ef30 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb45c0b1360 con 0x7fb45c13ef30 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb454001d90 con 0x7fb45c13ef30 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 10 monclient(hunting): dump: 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.269 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3072130285 0 0) 0x7fb45c0b11d0 con 0x7fb45c0b0860 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb440004eb0 con 0x7fb45c0b0860 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4096528571 0 0) 0x7fb45c1d64a0 con 0x7fb45c0af8e0 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb45c0b11d0 con 0x7fb45c0af8e0 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb44c001810 con 0x7fb45c0b0860 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:54.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 10 monclient(hunting): dump: 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb4580015d0 con 0x7fb45c0af8e0 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.278+0000 7fb452ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:54.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 10 monclient(hunting): dump: 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 27527748 0 0) 0x7fb440004eb0 con 0x7fb45c0b0860 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 >> v1:172.21.15.114:6790/0 conn(0x7fb45c0af8e0 legacy=0x7fb45c0b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 >> v1:172.21.15.176:6789/0 conn(0x7fb45c13ef30 legacy=0x7fb45c1dd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:54.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 1 monclient: found mon.a 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c0b2b20 con 0x7fb45c0b0860 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 10 monclient: _finish_auth 0 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb452ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb46457b640 5 monclient: authenticate success, global_id 4479 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb46457b640 10 monclient: _renew_subs 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb46457b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c1d7040 con 0x7fb45c0b0860 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb46457b640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb46457b640 10 monclient: _renew_subs 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb46457b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:45:54.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45c06de20 con 0x7fb45c0b0860 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.279+0000 7fb46457b640 1 librados: init done 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.280+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb44c0047e0 con 0x7fb45c0b0860 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.280+0000 7fb452ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.280+0000 7fb460b14640 4 set_mon_vals no callback set 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.280+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb44c001970 con 0x7fb45c0b0860 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.280+0000 7fb452ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.280+0000 7fb452ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.280+0000 7fb452ffd640 10 monclient: dump: 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:54.274 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.281+0000 7fb46457b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.281+0000 7fb46457b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.282+0000 7fb46457b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.282+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb444005080 con 0x7fb45c0b0860 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.282+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fb44c01a960 con 0x7fb45c0b0860 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.283+0000 7fb462317640 10 monclient: get_auth_request con 0x7fb44006e170 auth_method 0 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.283+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fb44c084950 con 0x7fb45c0b0860 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.283+0000 7fb452ffd640 10 client.4479.objecter ms_dispatch 0x7fb45c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.283+0000 7fb452ffd640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.283+0000 7fb452ffd640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:54.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.283+0000 7fb452ffd640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-29T10:45:54.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.283+0000 7fb452ffd640 10 client.4479.objecter ms_handle_connect 0x7fb44006e170 2024-03-29T10:45:54.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.296+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb44c050230 con 0x7fb45c0b0860 2024-03-29T10:45:54.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.296+0000 7fb452ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:54.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.296+0000 7fb452ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:54.646 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.664+0000 7fb46457b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:54.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.664+0000 7fb46457b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:54.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.664+0000 7fb46457b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:45:54.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.664+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb444005330 con 0x7fb45c0b0860 2024-03-29T10:45:54.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.666+0000 7fb452ffd640 1 -- 172.21.15.114:0/1793561930 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fb44c052160 con 0x7fb45c0b0860 2024-03-29T10:45:54.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.666+0000 7fb452ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:54.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.666+0000 7fb452ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:54.648 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:45:54.648 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:54.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 10 librados: watch_flush enter 2024-03-29T10:45:54.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 10 librados: watch_flush exit 2024-03-29T10:45:54.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-29T10:45:54.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 10 client.4479.objecter successfully canceled tick 2024-03-29T10:45:54.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 >> v1:172.21.15.114:6816/775044307 conn(0x7fb44006e170 legacy=0x7fb440070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:54.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 10 monclient: shutdown 2024-03-29T10:45:54.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:54.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 >> v1:172.21.15.114:6789/0 conn(0x7fb45c0b0860 legacy=0x7fb45c1d6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:54.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 shutdown_connections 2024-03-29T10:45:54.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 >> 172.21.15.114:0/1793561930 conn(0x7fb45c06b660 msgr2=0x7fb45c0ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:54.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 shutdown_connections 2024-03-29T10:45:54.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 1 -- 172.21.15.114:0/1793561930 wait complete. 2024-03-29T10:45:54.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:45:54.669+0000 7fb46457b640 1 librados: shutdown 2024-03-29T10:45:54.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:01.473 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:01.618 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.636+0000 7f136730e640 10 monclient: get_monmap_and_config 2024-03-29T10:46:01.618 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.636+0000 7f136730e640 10 monclient: build_initial_monmap 2024-03-29T10:46:01.618 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.636+0000 7f136730e640 10 monclient: monmap: 2024-03-29T10:46:01.618 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:01.618 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:01.618 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:46:01.636395+0000 2024-03-29T10:46:01.618 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:46:01.636395+0000 2024-03-29T10:46:01.619 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:01.619 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:01.619 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:01.619 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:46:01.619 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:46:01.619 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:01.620 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.636+0000 7f136730e640 1 Processor -- start 2024-03-29T10:46:01.620 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.637+0000 7f136730e640 1 -- start start 2024-03-29T10:46:01.620 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.637+0000 7f136730e640 10 monclient: init 2024-03-29T10:46:01.620 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.637+0000 7f136730e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:01.620 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.638+0000 7f136730e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:01.620 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.638+0000 7f136730e640 10 monclient(hunting): picked mon.noname-b con 0x7f13601457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:01.620 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.638+0000 7f13650aa640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f13601457f0 0x7f136013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:54342/0 (socket says 172.21.15.114:54342) 2024-03-29T10:46:01.620 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.638+0000 7f13650aa640 1 -- 172.21.15.114:0/8931177 learned_addr learned my addr 172.21.15.114:0/8931177 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:01.620 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.638+0000 7f136730e640 10 monclient(hunting): picked mon.noname-c con 0x7f136013f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:01.621 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.638+0000 7f136730e640 10 monclient(hunting): picked mon.noname-a con 0x7f13600a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:01.621 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.638+0000 7f136730e640 1 -- 172.21.15.114:0/8931177 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13600aaef0 con 0x7f13600a9d70 2024-03-29T10:46:01.621 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.638+0000 7f136730e640 1 -- 172.21.15.114:0/8931177 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1360145bd0 con 0x7f136013f650 2024-03-29T10:46:01.621 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.638+0000 7f136730e640 1 -- 172.21.15.114:0/8931177 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1360145eb0 con 0x7f13601457f0 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f136730e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f136730e640 10 monclient(hunting): authenticate will time out at 1226.879065s 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 1 -- 172.21.15.114:0/8931177 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234838466 0 0) 0x7f1360145eb0 con 0x7f13601457f0 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 1 -- 172.21.15.114:0/8931177 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134c003520 con 0x7f13601457f0 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 1 -- 172.21.15.114:0/8931177 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520844114 0 0) 0x7f1360145bd0 con 0x7f136013f650 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:46:01.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 1 -- 172.21.15.114:0/8931177 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1360145eb0 con 0x7f136013f650 2024-03-29T10:46:01.623 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 1 -- 172.21.15.114:0/8931177 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475480738 0 0) 0x7f134c003520 con 0x7f13601457f0 2024-03-29T10:46:01.623 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 1 -- 172.21.15.114:0/8931177 >> v1:172.21.15.176:6789/0 conn(0x7f136013f650 legacy=0x7f13600b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.623 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.639+0000 7f1357fff640 1 -- 172.21.15.114:0/8931177 >> v1:172.21.15.114:6789/0 conn(0x7f13600a9d70 legacy=0x7f136014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:01.624 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:01.624 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 1 monclient: found mon.noname-b 2024-03-29T10:46:01.624 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:01.624 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:46:01.624 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 1 -- 172.21.15.114:0/8931177 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1360146340 con 0x7f13601457f0 2024-03-29T10:46:01.624 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:01.624 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:01.624 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:01.624 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 1 -- 172.21.15.114:0/8931177 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f135c001690 con 0x7f13601457f0 2024-03-29T10:46:01.624 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f1357fff640 10 monclient: dump: 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:01.625 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f136730e640 5 monclient: authenticate success, global_id 4523 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.640+0000 7f136730e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.641+0000 7f1357fff640 1 -- 172.21.15.114:0/8931177 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f135c0017f0 con 0x7f13601457f0 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.641+0000 7f1357fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.641+0000 7f1357fff640 1 -- 172.21.15.114:0/8931177 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f135c001c00 con 0x7f13601457f0 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.641+0000 7f1357fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.641+0000 7f1357fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.641+0000 7f1357fff640 10 monclient: dump: 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:01.626 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:01.627 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:01.627 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:01.627 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:01.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.641+0000 7f136730e640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:01.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.641+0000 7f136730e640 4 set_mon_vals no callback set 2024-03-29T10:46:01.629 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.642+0000 7f136730e640 10 monclient: shutdown 2024-03-29T10:46:01.629 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.642+0000 7f136730e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:01.629 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.642+0000 7f136730e640 1 -- 172.21.15.114:0/8931177 >> v1:172.21.15.114:6790/0 conn(0x7f13601457f0 legacy=0x7f136013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.629 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.643+0000 7f136730e640 1 -- 172.21.15.114:0/8931177 shutdown_connections 2024-03-29T10:46:01.629 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.643+0000 7f136730e640 1 -- 172.21.15.114:0/8931177 >> 172.21.15.114:0/8931177 conn(0x7f1360066a00 msgr2=0x7f1360066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:01.629 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.643+0000 7f136730e640 1 -- 172.21.15.114:0/8931177 shutdown_connections 2024-03-29T10:46:01.629 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.643+0000 7f136730e640 1 -- 172.21.15.114:0/8931177 wait complete. 2024-03-29T10:46:01.629 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.644+0000 7f136730e640 10 monclient: build_initial_monmap 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.644+0000 7f136730e640 10 monclient: monmap: 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.644+0000 7f136730e640 1 librados: starting msgr at 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.644+0000 7f136730e640 1 librados: starting objecter 2024-03-29T10:46:01.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.644+0000 7f136730e640 1 Processor -- start 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.645+0000 7f136730e640 1 -- start start 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.645+0000 7f136730e640 1 librados: setting wanted keys 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.645+0000 7f136730e640 1 librados: calling monclient init 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.645+0000 7f136730e640 10 monclient: init 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f136730e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f136730e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f136730e640 10 monclient(hunting): picked mon.noname-a con 0x7f13600a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f136730e640 10 monclient(hunting): picked mon.noname-c con 0x7f136013f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f136730e640 10 monclient(hunting): picked mon.noname-b con 0x7f13601457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f136730e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1360135770 con 0x7f13600a9d70 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f13658ab640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f13601457f0 0x7f1360135060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:58762/0 (socket says 172.21.15.114:58762) 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f13658ab640 1 -- 172.21.15.114:0/111232540 learned_addr learned my addr 172.21.15.114:0/111232540 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1360137460 con 0x7f13601457f0 2024-03-29T10:46:01.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f1355ffb640 10 client.?.objecter ms_handle_connect 0x7f13601457f0 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f1355ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1360137740 con 0x7f136013f650 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f136730e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f136730e640 10 monclient(hunting): authenticate will time out at 1226.886572s 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180217222 0 0) 0x7f1360137460 con 0x7f13601457f0 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f1355ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f1355ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f1355ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f1355ffb640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1344004eb0 con 0x7f13601457f0 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.646+0000 7f1355ffb640 10 client.?.objecter ms_handle_connect 0x7f136013f650 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1662873200 0 0) 0x7f1344004eb0 con 0x7f13601457f0 2024-03-29T10:46:01.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1360137460 con 0x7f13601457f0 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1358001c60 con 0x7f13601457f0 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:01.633 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:01.634 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:01.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:01.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411628909 0 0) 0x7f1360137740 con 0x7f136013f650 2024-03-29T10:46:01.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:01.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:01.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:01.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:46:01.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.647+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1344004eb0 con 0x7f136013f650 2024-03-29T10:46:01.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 938408195 0 0) 0x7f1344004eb0 con 0x7f136013f650 2024-03-29T10:46:01.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1344005990 con 0x7f136013f650 2024-03-29T10:46:01.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1350001870 con 0x7f136013f650 2024-03-29T10:46:01.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:01.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:01.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:01.636 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 10 client.?.objecter ms_handle_connect 0x7f13600a9d70 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4108851933 0 0) 0x7f1344005990 con 0x7f136013f650 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 >> v1:172.21.15.176:6789/0 conn(0x7f13601457f0 legacy=0x7f1360135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 >> v1:172.21.15.114:6789/0 conn(0x7f13600a9d70 legacy=0x7f1360142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 1 monclient: found mon.c 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1360142c60 con 0x7f136013f650 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f1355ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f136730e640 5 monclient: authenticate success, global_id 4526 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f136730e640 10 monclient: _renew_subs 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f136730e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1360142e50 con 0x7f136013f650 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f136730e640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f136730e640 10 monclient: _renew_subs 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f136730e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:01.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13601daae0 con 0x7f136013f650 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.648+0000 7f136730e640 1 librados: init done 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.649+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13500035f0 con 0x7f136013f650 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.649+0000 7f1355ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.649+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f1350003760 con 0x7f136013f650 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.649+0000 7f1355ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.649+0000 7f1355ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.649+0000 7f1355ffb640 10 monclient: dump: 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:01.639 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.649+0000 7f13577fe640 4 set_mon_vals no callback set 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.650+0000 7f136730e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.650+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f1350019730 con 0x7f136013f650 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.650+0000 7f136730e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.650+0000 7f136730e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.650+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1360075f10 con 0x7f136013f650 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.651+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f13500835e0 con 0x7f136013f650 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.651+0000 7f1355ffb640 10 client.4526.objecter ms_dispatch 0x7f1360000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.651+0000 7f1355ffb640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.651+0000 7f1355ffb640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.651+0000 7f13650aa640 10 monclient: get_auth_request con 0x7f134406e0c0 auth_method 0 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.651+0000 7f1355ffb640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T10:46:01.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.652+0000 7f1355ffb640 10 client.4526.objecter ms_handle_connect 0x7f134406e0c0 2024-03-29T10:46:01.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.660+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f135004ef20 con 0x7f136013f650 2024-03-29T10:46:01.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.660+0000 7f1355ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:01.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.660+0000 7f1355ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:01.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.975+0000 7f136730e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:01.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.975+0000 7f136730e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:01.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.975+0000 7f136730e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:01.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.975+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13600b3680 con 0x7f136013f650 2024-03-29T10:46:01.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.976+0000 7f1355ffb640 1 -- 172.21.15.114:0/111232540 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f1350082c90 con 0x7f136013f650 2024-03-29T10:46:01.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.976+0000 7f1355ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:01.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.976+0000 7f1355ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:01.960 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:46:01.960 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:01.964 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.982+0000 7f136730e640 10 librados: watch_flush enter 2024-03-29T10:46:01.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.982+0000 7f136730e640 10 librados: watch_flush exit 2024-03-29T10:46:01.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.982+0000 7f136730e640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T10:46:01.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.982+0000 7f136730e640 10 client.4526.objecter successfully canceled tick 2024-03-29T10:46:01.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.983+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 >> v1:172.21.15.114:6816/775044307 conn(0x7f134406e0c0 legacy=0x7f1344070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.983+0000 7f136730e640 10 monclient: shutdown 2024-03-29T10:46:01.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.983+0000 7f136730e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:01.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.983+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 >> v1:172.21.15.114:6790/0 conn(0x7f136013f650 legacy=0x7f1360134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.983+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 shutdown_connections 2024-03-29T10:46:01.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.983+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 >> 172.21.15.114:0/111232540 conn(0x7f1360066a00 msgr2=0x7f1360066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:01.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.983+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 shutdown_connections 2024-03-29T10:46:01.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.983+0000 7f136730e640 1 -- 172.21.15.114:0/111232540 wait complete. 2024-03-29T10:46:01.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:01.983+0000 7f136730e640 1 librados: shutdown 2024-03-29T10:46:01.982 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:08.789 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:08.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.964+0000 7ff978ede640 10 monclient: get_monmap_and_config 2024-03-29T10:46:08.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.964+0000 7ff978ede640 10 monclient: build_initial_monmap 2024-03-29T10:46:08.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.964+0000 7ff978ede640 10 monclient: monmap: 2024-03-29T10:46:08.948 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:08.948 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:08.948 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:46:08.965904+0000 2024-03-29T10:46:08.948 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:46:08.965904+0000 2024-03-29T10:46:08.948 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:08.949 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:08.949 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:08.949 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:46:08.949 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:46:08.949 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:08.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.965+0000 7ff978ede640 1 Processor -- start 2024-03-29T10:46:08.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.965+0000 7ff978ede640 1 -- start start 2024-03-29T10:46:08.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.965+0000 7ff978ede640 10 monclient: init 2024-03-29T10:46:08.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.966+0000 7ff978ede640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:08.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.966+0000 7ff978ede640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:08.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.966+0000 7ff978ede640 10 monclient(hunting): picked mon.noname-c con 0x7ff9740b4820 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:08.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.966+0000 7ff978ede640 10 monclient(hunting): picked mon.noname-a con 0x7ff9740a9ce0 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:08.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.966+0000 7ff978ede640 10 monclient(hunting): picked mon.noname-b con 0x7ff9740b3cd0 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:08.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.966+0000 7ff978ede640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9740b4180 con 0x7ff9740a9ce0 2024-03-29T10:46:08.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.966+0000 7ff978ede640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9740aaf60 con 0x7ff9740b4820 2024-03-29T10:46:08.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.966+0000 7ff978ede640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97413c200 con 0x7ff9740b3cd0 2024-03-29T10:46:08.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.966+0000 7ff978ede640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:08.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.966+0000 7ff978ede640 10 monclient(hunting): authenticate will time out at 1234.207663s 2024-03-29T10:46:08.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.967+0000 7ff973fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff9740b3cd0 0x7ff97414d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:46646/0 (socket says 172.21.15.114:46646) 2024-03-29T10:46:08.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.967+0000 7ff973fff640 1 -- 172.21.15.114:0/394591472 learned_addr learned my addr 172.21.15.114:0/394591472 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:08.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.967+0000 7ff9727fc640 1 -- 172.21.15.114:0/394591472 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338691978 0 0) 0x7ff97413c200 con 0x7ff9740b3cd0 2024-03-29T10:46:08.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.967+0000 7ff9727fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:08.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.967+0000 7ff9727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:08.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.967+0000 7ff9727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:08.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.967+0000 7ff9727fc640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:46:08.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.967+0000 7ff9727fc640 1 -- 172.21.15.114:0/394591472 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff960003520 con 0x7ff9740b3cd0 2024-03-29T10:46:08.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.967+0000 7ff9727fc640 1 -- 172.21.15.114:0/394591472 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4286682826 0 0) 0x7ff960003520 con 0x7ff9740b3cd0 2024-03-29T10:46:08.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.967+0000 7ff9727fc640 1 -- 172.21.15.114:0/394591472 >> v1:172.21.15.176:6789/0 conn(0x7ff9740b4820 legacy=0x7ff97413baf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:08.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.968+0000 7ff9727fc640 1 -- 172.21.15.114:0/394591472 >> v1:172.21.15.114:6789/0 conn(0x7ff9740a9ce0 legacy=0x7ff974145770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:08.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:08.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 1 monclient: found mon.noname-b 2024-03-29T10:46:08.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:08.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:46:08.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 1 -- 172.21.15.114:0/394591472 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff974145e80 con 0x7ff9740b3cd0 2024-03-29T10:46:08.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:08.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:08.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:08.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 1 -- 172.21.15.114:0/394591472 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff964001670 con 0x7ff9740b3cd0 2024-03-29T10:46:08.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:08.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:08.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.971+0000 7ff9727fc640 10 monclient: dump: 2024-03-29T10:46:08.958 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:08.958 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.972+0000 7ff978ede640 5 monclient: authenticate success, global_id 4529 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.972+0000 7ff978ede640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.972+0000 7ff9727fc640 1 -- 172.21.15.114:0/394591472 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9640018d0 con 0x7ff9740b3cd0 2024-03-29T10:46:08.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.972+0000 7ff9727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:08.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.972+0000 7ff978ede640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:08.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.972+0000 7ff978ede640 4 set_mon_vals no callback set 2024-03-29T10:46:08.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.973+0000 7ff978ede640 10 monclient: shutdown 2024-03-29T10:46:08.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.973+0000 7ff978ede640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:08.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.973+0000 7ff978ede640 1 -- 172.21.15.114:0/394591472 >> v1:172.21.15.114:6790/0 conn(0x7ff9740b3cd0 legacy=0x7ff97414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:08.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.973+0000 7ff9727fc640 1 -- 172.21.15.114:0/394591472 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff964001d20 con 0x7ff9740b3cd0 2024-03-29T10:46:08.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.973+0000 7ff9727fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:46:08.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.973+0000 7ff978ede640 1 -- 172.21.15.114:0/394591472 shutdown_connections 2024-03-29T10:46:08.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.973+0000 7ff978ede640 1 -- 172.21.15.114:0/394591472 >> 172.21.15.114:0/394591472 conn(0x7ff97406d070 msgr2=0x7ff974066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:08.961 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.974+0000 7ff978ede640 1 -- 172.21.15.114:0/394591472 shutdown_connections 2024-03-29T10:46:08.961 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.976+0000 7ff978ede640 1 -- 172.21.15.114:0/394591472 wait complete. 2024-03-29T10:46:08.961 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.977+0000 7ff978ede640 10 monclient: build_initial_monmap 2024-03-29T10:46:08.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.977+0000 7ff978ede640 10 monclient: monmap: 2024-03-29T10:46:08.964 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:08.964 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:08.964 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:46:08.964 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:46:08.964 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:08.964 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:08.964 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:08.964 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:46:08.964 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:46:08.964 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:08.964 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.977+0000 7ff978ede640 1 librados: starting msgr at 2024-03-29T10:46:08.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.977+0000 7ff978ede640 1 librados: starting objecter 2024-03-29T10:46:08.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.977+0000 7ff978ede640 1 Processor -- start 2024-03-29T10:46:08.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.977+0000 7ff978ede640 1 -- start start 2024-03-29T10:46:08.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.977+0000 7ff978ede640 1 librados: setting wanted keys 2024-03-29T10:46:08.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.977+0000 7ff978ede640 1 librados: calling monclient init 2024-03-29T10:46:08.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.977+0000 7ff978ede640 10 monclient: init 2024-03-29T10:46:08.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.978+0000 7ff978ede640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:08.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.978+0000 7ff978ede640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:08.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff978ede640 10 monclient(hunting): picked mon.noname-b con 0x7ff9740a9ce0 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff978ede640 10 monclient(hunting): picked mon.noname-c con 0x7ff9740b3cd0 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff978ede640 10 monclient(hunting): picked mon.noname-a con 0x7ff9740b4820 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff978ede640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97413ed50 con 0x7ff9740b4820 2024-03-29T10:46:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff978ede640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97413e900 con 0x7ff9740a9ce0 2024-03-29T10:46:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff978ede640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97413ea90 con 0x7ff9740b3cd0 2024-03-29T10:46:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff978ede640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff978ede640 10 monclient(hunting): authenticate will time out at 1234.222269s 2024-03-29T10:46:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff9737fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ff9740a9ce0 0x7ff97413e1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:60567/0 (socket says 172.21.15.114:60567) 2024-03-29T10:46:08.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff9737fe640 1 -- 172.21.15.114:0/3870496755 learned_addr learned my addr 172.21.15.114:0/3870496755 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:08.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff96bfff640 10 client.?.objecter ms_handle_connect 0x7ff9740a9ce0 2024-03-29T10:46:08.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.981+0000 7ff96bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:08.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 client.?.objecter ms_handle_connect 0x7ff9740b3cd0 2024-03-29T10:46:08.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:08.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951685926 0 0) 0x7ff97413e900 con 0x7ff9740a9ce0 2024-03-29T10:46:08.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:08.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:08.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:08.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:46:08.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff954004eb0 con 0x7ff9740a9ce0 2024-03-29T10:46:08.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656221376 0 0) 0x7ff97413ea90 con 0x7ff9740b3cd0 2024-03-29T10:46:08.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:08.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:08.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:08.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): my global_id is 4532 2024-03-29T10:46:08.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff97413e900 con 0x7ff9740b3cd0 2024-03-29T10:46:08.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1465777619 0 0) 0x7ff954004eb0 con 0x7ff9740a9ce0 2024-03-29T10:46:08.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff97413ea90 con 0x7ff9740a9ce0 2024-03-29T10:46:08.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3307318459 0 0) 0x7ff97413e900 con 0x7ff9740b3cd0 2024-03-29T10:46:08.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff954004eb0 con 0x7ff9740b3cd0 2024-03-29T10:46:08.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff96c001b50 con 0x7ff9740a9ce0 2024-03-29T10:46:08.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:08.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:08.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): dump: 2024-03-29T10:46:08.971 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:08.972 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:08.972 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:08.972 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:08.972 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:08.972 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:08.972 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:08.972 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:08.973 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:08.973 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff95c0017b0 con 0x7ff9740b3cd0 2024-03-29T10:46:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.982+0000 7ff96bfff640 10 monclient(hunting): dump: 2024-03-29T10:46:08.973 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:08.974 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:08.974 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:08.974 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:08.974 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:08.974 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:08.974 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:08.974 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:08.974 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:08.974 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:08.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.983+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4027138396 0 0) 0x7ff97413ea90 con 0x7ff9740a9ce0 2024-03-29T10:46:08.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.983+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 >> v1:172.21.15.114:6790/0 conn(0x7ff9740b3cd0 legacy=0x7ff9741de6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:08.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.983+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 >> v1:172.21.15.114:6789/0 conn(0x7ff9740b4820 legacy=0x7ff9740715b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:08.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.984+0000 7ff96bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:08.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.984+0000 7ff96bfff640 1 monclient: found mon.b 2024-03-29T10:46:08.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.984+0000 7ff96bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:08.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.984+0000 7ff96bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:08.975 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.984+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff974144ac0 con 0x7ff9740a9ce0 2024-03-29T10:46:08.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff96bfff640 10 monclient: _finish_auth 0 2024-03-29T10:46:08.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff96bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:08.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff96bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:08.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff978ede640 5 monclient: authenticate success, global_id 4459 2024-03-29T10:46:08.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff978ede640 10 monclient: _renew_subs 2024-03-29T10:46:08.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff978ede640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:08.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff978ede640 1 -- 172.21.15.114:0/3870496755 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff974144d10 con 0x7ff9740a9ce0 2024-03-29T10:46:08.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff978ede640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:08.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff978ede640 10 monclient: _renew_subs 2024-03-29T10:46:08.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff978ede640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:08.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff978ede640 1 -- 172.21.15.114:0/3870496755 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff974071ea0 con 0x7ff9740a9ce0 2024-03-29T10:46:08.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.985+0000 7ff978ede640 1 librados: init done 2024-03-29T10:46:08.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.986+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff96c001cb0 con 0x7ff9740a9ce0 2024-03-29T10:46:08.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.986+0000 7ff96bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:08.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.986+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff96c004d80 con 0x7ff9740a9ce0 2024-03-29T10:46:08.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.986+0000 7ff96bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:08.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.986+0000 7ff96bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:08.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.986+0000 7ff96bfff640 10 monclient: dump: 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.986+0000 7ff971ffb640 4 set_mon_vals no callback set 2024-03-29T10:46:08.981 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.987+0000 7ff969ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:08.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.987+0000 7ff969ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:08.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.987+0000 7ff969ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:08.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.987+0000 7ff969ffb640 1 -- 172.21.15.114:0/3870496755 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff97406ef20 con 0x7ff9740a9ce0 2024-03-29T10:46:08.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.987+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7ff96c004ee0 con 0x7ff9740a9ce0 2024-03-29T10:46:08.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.988+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7ff96c084d60 con 0x7ff9740a9ce0 2024-03-29T10:46:08.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.988+0000 7ff96bfff640 10 client.4459.objecter ms_dispatch 0x7ff974000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:08.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.988+0000 7ff96bfff640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:08.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.988+0000 7ff96bfff640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:08.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.988+0000 7ff96bfff640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T10:46:08.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.988+0000 7ff972ffd640 10 monclient: get_auth_request con 0x7ff95406df70 auth_method 0 2024-03-29T10:46:08.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.988+0000 7ff96bfff640 10 client.4459.objecter ms_handle_connect 0x7ff95406df70 2024-03-29T10:46:08.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.995+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff96c050640 con 0x7ff9740a9ce0 2024-03-29T10:46:08.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.995+0000 7ff96bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:08.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:08.995+0000 7ff96bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:09.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.323+0000 7ff969ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:09.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.323+0000 7ff969ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:09.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.323+0000 7ff969ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:09.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.323+0000 7ff969ffb640 1 -- 172.21.15.114:0/3870496755 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff97404f730 con 0x7ff9740a9ce0 2024-03-29T10:46:09.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.327+0000 7ff96bfff640 1 -- 172.21.15.114:0/3870496755 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7ff96c052570 con 0x7ff9740a9ce0 2024-03-29T10:46:09.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.327+0000 7ff96bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:09.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.327+0000 7ff96bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:09.311 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:46:09.311 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:09.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.332+0000 7ff969ffb640 10 librados: watch_flush enter 2024-03-29T10:46:09.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.332+0000 7ff969ffb640 10 librados: watch_flush exit 2024-03-29T10:46:09.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.332+0000 7ff969ffb640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T10:46:09.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.332+0000 7ff969ffb640 10 client.4459.objecter successfully canceled tick 2024-03-29T10:46:09.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.333+0000 7ff969ffb640 1 -- 172.21.15.114:0/3870496755 >> v1:172.21.15.114:6816/775044307 conn(0x7ff95406df70 legacy=0x7ff954070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:09.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.333+0000 7ff969ffb640 10 monclient: shutdown 2024-03-29T10:46:09.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.333+0000 7ff969ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:09.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.333+0000 7ff969ffb640 1 -- 172.21.15.114:0/3870496755 >> v1:172.21.15.176:6789/0 conn(0x7ff9740a9ce0 legacy=0x7ff97413e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:09.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.333+0000 7ff969ffb640 1 -- 172.21.15.114:0/3870496755 shutdown_connections 2024-03-29T10:46:09.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.333+0000 7ff969ffb640 1 -- 172.21.15.114:0/3870496755 >> 172.21.15.114:0/3870496755 conn(0x7ff97406d070 msgr2=0x7ff9740b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:09.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.333+0000 7ff969ffb640 1 -- 172.21.15.114:0/3870496755 shutdown_connections 2024-03-29T10:46:09.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.334+0000 7ff969ffb640 1 -- 172.21.15.114:0/3870496755 wait complete. 2024-03-29T10:46:09.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:09.334+0000 7ff969ffb640 1 librados: shutdown 2024-03-29T10:46:09.334 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:16.140 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:16.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.302+0000 7f51da15c640 10 monclient: get_monmap_and_config 2024-03-29T10:46:16.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.302+0000 7f51da15c640 10 monclient: build_initial_monmap 2024-03-29T10:46:16.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.302+0000 7f51da15c640 10 monclient: monmap: 2024-03-29T10:46:16.361 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:16.361 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:16.361 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:46:16.303366+0000 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:46:16.303366+0000 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.302+0000 7f51da15c640 1 Processor -- start 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.302+0000 7f51da15c640 1 -- start start 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.302+0000 7f51da15c640 10 monclient: init 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.303+0000 7f51da15c640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.303+0000 7f51da15c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.303+0000 7f51da15c640 10 monclient(hunting): picked mon.noname-b con 0x7f51d40ad6c0 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:16.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.303+0000 7f51da15c640 10 monclient(hunting): picked mon.noname-a con 0x7f51d40af6c0 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.303+0000 7f51da15c640 10 monclient(hunting): picked mon.noname-c con 0x7f51d414fdc0 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.303+0000 7f51da15c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40a94e0 con 0x7f51d40af6c0 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.303+0000 7f51da15c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40af3a0 con 0x7f51d414fdc0 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.303+0000 7f51da15c640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40afb60 con 0x7f51d40ad6c0 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.303+0000 7f51da15c640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.303+0000 7f51da15c640 10 monclient(hunting): authenticate will time out at 1241.544720s 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.306+0000 7f51d3fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f51d414fdc0 0x7f51d415a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:52275/0 (socket says 172.21.15.114:52275) 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.306+0000 7f51d3fff640 1 -- 172.21.15.114:0/1245059497 learned_addr learned my addr 172.21.15.114:0/1245059497 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.308+0000 7f51d27fc640 1 -- 172.21.15.114:0/1245059497 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319040270 0 0) 0x7f51d40af3a0 con 0x7f51d414fdc0 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.308+0000 7f51d27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.308+0000 7f51d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.308+0000 7f51d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.308+0000 7f51d27fc640 10 monclient(hunting): my global_id is 4462 2024-03-29T10:46:16.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.308+0000 7f51d27fc640 1 -- 172.21.15.114:0/1245059497 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51bc003520 con 0x7f51d414fdc0 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.308+0000 7f51d27fc640 1 -- 172.21.15.114:0/1245059497 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4283426713 0 0) 0x7f51bc003520 con 0x7f51d414fdc0 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 1 -- 172.21.15.114:0/1245059497 >> v1:172.21.15.114:6790/0 conn(0x7f51d40ad6c0 legacy=0x7f51d414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 1 -- 172.21.15.114:0/1245059497 >> v1:172.21.15.114:6789/0 conn(0x7f51d40af6c0 legacy=0x7f51d4152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 1 monclient: found mon.noname-c 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 1 -- 172.21.15.114:0/1245059497 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40aab80 con 0x7f51d414fdc0 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 1 -- 172.21.15.114:0/1245059497 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f51cc001a10 con 0x7f51d414fdc0 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:16.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 10 monclient: dump: 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51da15c640 5 monclient: authenticate success, global_id 4462 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51da15c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 1 -- 172.21.15.114:0/1245059497 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51cc001c70 con 0x7f51d414fdc0 2024-03-29T10:46:16.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 1 -- 172.21.15.114:0/1245059497 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f51cc002080 con 0x7f51d414fdc0 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51da15c640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51da15c640 4 set_mon_vals no callback set 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.378+0000 7f51d27fc640 10 monclient: dump: 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:16.366 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:16.367 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:16.367 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:16.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.379+0000 7f51da15c640 10 monclient: shutdown 2024-03-29T10:46:16.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.379+0000 7f51da15c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:16.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.379+0000 7f51da15c640 1 -- 172.21.15.114:0/1245059497 >> v1:172.21.15.176:6789/0 conn(0x7f51d414fdc0 legacy=0x7f51d415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.379+0000 7f51da15c640 1 -- 172.21.15.114:0/1245059497 shutdown_connections 2024-03-29T10:46:16.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.379+0000 7f51da15c640 1 -- 172.21.15.114:0/1245059497 >> 172.21.15.114:0/1245059497 conn(0x7f51d40b35e0 msgr2=0x7f51d40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:16.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.379+0000 7f51da15c640 1 -- 172.21.15.114:0/1245059497 shutdown_connections 2024-03-29T10:46:16.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.379+0000 7f51da15c640 1 -- 172.21.15.114:0/1245059497 wait complete. 2024-03-29T10:46:16.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.380+0000 7f51da15c640 10 monclient: build_initial_monmap 2024-03-29T10:46:16.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.380+0000 7f51da15c640 10 monclient: monmap: 2024-03-29T10:46:16.374 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:16.374 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.380+0000 7f51da15c640 1 librados: starting msgr at 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.380+0000 7f51da15c640 1 librados: starting objecter 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.380+0000 7f51da15c640 1 Processor -- start 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.381+0000 7f51da15c640 1 -- start start 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.381+0000 7f51da15c640 1 librados: setting wanted keys 2024-03-29T10:46:16.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.381+0000 7f51da15c640 1 librados: calling monclient init 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.381+0000 7f51da15c640 10 monclient: init 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.381+0000 7f51da15c640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.381+0000 7f51da15c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.381+0000 7f51da15c640 10 monclient(hunting): picked mon.noname-c con 0x7f51d40ad6c0 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.384+0000 7f51da15c640 10 monclient(hunting): picked mon.noname-b con 0x7f51d40af6c0 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.384+0000 7f51d37fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f51d40ad6c0 0x7f51d4151510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:49120/0 (socket says 172.21.15.114:49120) 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.384+0000 7f51d2ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f51d40af6c0 0x7f51d41e6490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:52781/0 (socket says 172.21.15.114:52781) 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.385+0000 7f51da15c640 10 monclient(hunting): picked mon.noname-a con 0x7f51d414fdc0 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.385+0000 7f51da15c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d41e5cd0 con 0x7f51d414fdc0 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.385+0000 7f51da15c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40722a0 con 0x7f51d40af6c0 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.385+0000 7f51da15c640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d4072580 con 0x7f51d40ad6c0 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.385+0000 7f51da15c640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.385+0000 7f51da15c640 10 monclient(hunting): authenticate will time out at 1241.626264s 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.385+0000 7f51d37fe640 1 -- 172.21.15.114:0/3397798249 learned_addr learned my addr 172.21.15.114:0/3397798249 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:16.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.385+0000 7f51bbfff640 10 client.?.objecter ms_handle_connect 0x7f51d40ad6c0 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.385+0000 7f51bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 10 client.?.objecter ms_handle_connect 0x7f51d40af6c0 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071085783 0 0) 0x7f51d4072580 con 0x7f51d40ad6c0 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c0004eb0 con 0x7f51d40ad6c0 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34735536 0 0) 0x7f51d40722a0 con 0x7f51d40af6c0 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:16.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 10 monclient(hunting): my global_id is 4465 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.386+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c0006230 con 0x7f51d40af6c0 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.387+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304840916 0 0) 0x7f51c0006230 con 0x7f51d40af6c0 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.387+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51d40722a0 con 0x7f51d40af6c0 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.387+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f51c8001860 con 0x7f51d40af6c0 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.387+0000 7f51bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.387+0000 7f51bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.387+0000 7f51bbfff640 10 monclient(hunting): dump: 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:16.378 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.387+0000 7f51bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.387+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1657029637 0 0) 0x7f51c0004eb0 con 0x7f51d40ad6c0 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.387+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51c0006230 con 0x7f51d40ad6c0 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.387+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f51c4001970 con 0x7f51d40ad6c0 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.387+0000 7f51bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 10 monclient(hunting): dump: 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:16.379 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:16.380 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:16.380 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:16.380 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:16.380 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:16.380 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:16.380 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:16.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 10 client.?.objecter ms_handle_connect 0x7f51d414fdc0 2024-03-29T10:46:16.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:16.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2522276180 0 0) 0x7f51c0006230 con 0x7f51d40ad6c0 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 >> v1:172.21.15.176:6789/0 conn(0x7f51d40af6c0 legacy=0x7f51d41e6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 >> v1:172.21.15.114:6789/0 conn(0x7f51d414fdc0 legacy=0x7f51d408a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 1 monclient: found mon.c 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d41e5f70 con 0x7f51d40ad6c0 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 10 monclient: _finish_auth 0 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51da15c640 5 monclient: authenticate success, global_id 4538 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51da15c640 10 monclient: _renew_subs 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51da15c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:16.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51da15c640 1 -- 172.21.15.114:0/3397798249 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d41e6ba0 con 0x7f51d40ad6c0 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51da15c640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51da15c640 10 monclient: _renew_subs 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51da15c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51da15c640 1 -- 172.21.15.114:0/3397798249 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d414d360 con 0x7f51d40ad6c0 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.388+0000 7f51da15c640 1 librados: init done 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.389+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51c4001c90 con 0x7f51d40ad6c0 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.389+0000 7f51bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.389+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f51c4003a60 con 0x7f51d40ad6c0 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.389+0000 7f51bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.389+0000 7f51bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.389+0000 7f51bbfff640 10 monclient: dump: 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:16.382 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:16.383 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:16.383 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:16.383 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:16.383 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:16.383 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:16.383 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:16.383 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:16.383 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:16.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.391+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f51c4003c40 con 0x7f51d40ad6c0 2024-03-29T10:46:16.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.391+0000 7f51da15c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:16.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.391+0000 7f51da15c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:16.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.391+0000 7f51da15c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:16.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.391+0000 7f51da15c640 1 -- 172.21.15.114:0/3397798249 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d414d6b0 con 0x7f51d40ad6c0 2024-03-29T10:46:16.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.392+0000 7f51d2ffd640 10 monclient: get_auth_request con 0x7f51c006e210 auth_method 0 2024-03-29T10:46:16.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.392+0000 7f51bbfff640 10 client.4538.objecter ms_handle_connect 0x7f51c006e210 2024-03-29T10:46:16.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.392+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f51c40546a0 con 0x7f51d40ad6c0 2024-03-29T10:46:16.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.392+0000 7f51bbfff640 10 client.4538.objecter ms_dispatch 0x7f51d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:16.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.392+0000 7f51bbfff640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:16.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.392+0000 7f51bbfff640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:16.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.392+0000 7f51bbfff640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-29T10:46:16.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.392+0000 7f51d17fa640 4 set_mon_vals no callback set 2024-03-29T10:46:16.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.400+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51c4050940 con 0x7f51d40ad6c0 2024-03-29T10:46:16.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.400+0000 7f51bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:16.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.400+0000 7f51bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:16.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.738+0000 7f51da15c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:16.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.738+0000 7f51da15c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:16.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.738+0000 7f51da15c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:16.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.738+0000 7f51da15c640 1 -- 172.21.15.114:0/3397798249 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51d4075f10 con 0x7f51d40ad6c0 2024-03-29T10:46:16.725 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:46:16.726 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:16.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.742+0000 7f51bbfff640 1 -- 172.21.15.114:0/3397798249 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f51c4050ae0 con 0x7f51d40ad6c0 2024-03-29T10:46:16.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.742+0000 7f51bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:16.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.742+0000 7f51bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:16.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.745+0000 7f51da15c640 10 librados: watch_flush enter 2024-03-29T10:46:16.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.745+0000 7f51da15c640 10 librados: watch_flush exit 2024-03-29T10:46:16.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.745+0000 7f51da15c640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-29T10:46:16.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.745+0000 7f51da15c640 10 client.4538.objecter successfully canceled tick 2024-03-29T10:46:16.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.746+0000 7f51da15c640 1 -- 172.21.15.114:0/3397798249 >> v1:172.21.15.114:6816/775044307 conn(0x7f51c006e210 legacy=0x7f51c00706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.746+0000 7f51da15c640 10 monclient: shutdown 2024-03-29T10:46:16.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.746+0000 7f51da15c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:16.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.746+0000 7f51da15c640 1 -- 172.21.15.114:0/3397798249 >> v1:172.21.15.114:6790/0 conn(0x7f51d40ad6c0 legacy=0x7f51d4151510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.746+0000 7f51da15c640 1 -- 172.21.15.114:0/3397798249 shutdown_connections 2024-03-29T10:46:16.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.746+0000 7f51da15c640 1 -- 172.21.15.114:0/3397798249 >> 172.21.15.114:0/3397798249 conn(0x7f51d40b35e0 msgr2=0x7f51d40aeb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:16.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.747+0000 7f51da15c640 1 -- 172.21.15.114:0/3397798249 shutdown_connections 2024-03-29T10:46:16.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.747+0000 7f51da15c640 1 -- 172.21.15.114:0/3397798249 wait complete. 2024-03-29T10:46:16.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:16.747+0000 7f51da15c640 1 librados: shutdown 2024-03-29T10:46:16.746 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:23.553 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:23.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.722+0000 7fbbad4f8640 10 monclient: get_monmap_and_config 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.722+0000 7fbbad4f8640 10 monclient: build_initial_monmap 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.722+0000 7fbbad4f8640 10 monclient: monmap: 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:46:23.723327+0000 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:46:23.723327+0000 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:46:23.705 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:23.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.722+0000 7fbbad4f8640 1 Processor -- start 2024-03-29T10:46:23.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.723+0000 7fbbad4f8640 1 -- start start 2024-03-29T10:46:23.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.723+0000 7fbbad4f8640 10 monclient: init 2024-03-29T10:46:23.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbbad4f8640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:23.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbbad4f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:23.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbbad4f8640 10 monclient(hunting): picked mon.noname-b con 0x7fbba806c250 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:23.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbbad4f8640 10 monclient(hunting): picked mon.noname-c con 0x7fbba8072400 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:23.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbbad4f8640 10 monclient(hunting): picked mon.noname-a con 0x7fbba806c630 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:23.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbbad4f8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba806d760 con 0x7fbba806c630 2024-03-29T10:46:23.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbbad4f8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba8070500 con 0x7fbba8072400 2024-03-29T10:46:23.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbbad4f8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba80aff50 con 0x7fbba806c250 2024-03-29T10:46:23.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbbad4f8640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:23.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbbad4f8640 10 monclient(hunting): authenticate will time out at 1248.965188s 2024-03-29T10:46:23.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbba6ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fbba806c250 0x7fbba8071cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:49754/0 (socket says 172.21.15.114:49754) 2024-03-29T10:46:23.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbba6ffd640 1 -- 172.21.15.114:0/1327509650 learned_addr learned my addr 172.21.15.114:0/1327509650 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:23.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbba67fc640 1 -- 172.21.15.114:0/1327509650 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625599623 0 0) 0x7fbba80aff50 con 0x7fbba806c250 2024-03-29T10:46:23.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbba67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:23.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbba67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:23.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:23.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbba67fc640 10 monclient(hunting): my global_id is 4541 2024-03-29T10:46:23.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.724+0000 7fbba67fc640 1 -- 172.21.15.114:0/1327509650 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb94003520 con 0x7fbba806c250 2024-03-29T10:46:23.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.725+0000 7fbba67fc640 1 -- 172.21.15.114:0/1327509650 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 108979477 0 0) 0x7fbb94003520 con 0x7fbba806c250 2024-03-29T10:46:23.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.725+0000 7fbba67fc640 1 -- 172.21.15.114:0/1327509650 >> v1:172.21.15.176:6789/0 conn(0x7fbba8072400 legacy=0x7fbba81465b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:23.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.727+0000 7fbba67fc640 1 -- 172.21.15.114:0/1327509650 >> v1:172.21.15.114:6789/0 conn(0x7fbba806c630 legacy=0x7fbba8151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:23.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:23.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 1 monclient: found mon.noname-b 2024-03-29T10:46:23.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:23.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:46:23.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 1 -- 172.21.15.114:0/1327509650 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba80b00e0 con 0x7fbba806c250 2024-03-29T10:46:23.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:23.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:23.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:23.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 1 -- 172.21.15.114:0/1327509650 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fbb98001670 con 0x7fbba806c250 2024-03-29T10:46:23.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:23.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:23.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbba67fc640 10 monclient: dump: 2024-03-29T10:46:23.714 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:23.714 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:23.714 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:23.715 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:23.715 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:23.715 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:23.715 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:23.715 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:23.715 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:23.715 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:23.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbbad4f8640 5 monclient: authenticate success, global_id 4541 2024-03-29T10:46:23.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.729+0000 7fbbad4f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:23.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.731+0000 7fbba67fc640 1 -- 172.21.15.114:0/1327509650 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb980018d0 con 0x7fbba806c250 2024-03-29T10:46:23.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.731+0000 7fbba67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:23.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.731+0000 7fbba67fc640 1 -- 172.21.15.114:0/1327509650 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fbb98001d20 con 0x7fbba806c250 2024-03-29T10:46:23.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.731+0000 7fbba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:23.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.731+0000 7fbba67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:23.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.731+0000 7fbba67fc640 10 monclient: dump: 2024-03-29T10:46:23.718 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:23.719 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:23.719 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:23.719 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:23.719 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:23.719 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:23.719 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:23.719 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:23.719 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:23.719 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:23.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.732+0000 7fbbad4f8640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:23.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.732+0000 7fbbad4f8640 4 set_mon_vals no callback set 2024-03-29T10:46:23.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.733+0000 7fbbad4f8640 10 monclient: shutdown 2024-03-29T10:46:23.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.733+0000 7fbbad4f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:23.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.733+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1327509650 >> v1:172.21.15.114:6790/0 conn(0x7fbba806c250 legacy=0x7fbba8071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:23.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.734+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1327509650 shutdown_connections 2024-03-29T10:46:23.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.734+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1327509650 >> 172.21.15.114:0/1327509650 conn(0x7fbba8067610 msgr2=0x7fbba80b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:23.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.735+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1327509650 shutdown_connections 2024-03-29T10:46:23.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.735+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1327509650 wait complete. 2024-03-29T10:46:23.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.736+0000 7fbbad4f8640 10 monclient: build_initial_monmap 2024-03-29T10:46:23.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.736+0000 7fbbad4f8640 10 monclient: monmap: 2024-03-29T10:46:23.722 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:23.722 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:23.722 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:46:23.722 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:46:23.723 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:23.723 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:23.723 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:23.723 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:46:23.723 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:46:23.723 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:23.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.736+0000 7fbbad4f8640 1 librados: starting msgr at 2024-03-29T10:46:23.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.736+0000 7fbbad4f8640 1 librados: starting objecter 2024-03-29T10:46:23.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.736+0000 7fbbad4f8640 1 Processor -- start 2024-03-29T10:46:23.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.736+0000 7fbbad4f8640 1 -- start start 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.736+0000 7fbbad4f8640 1 librados: setting wanted keys 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.736+0000 7fbbad4f8640 1 librados: calling monclient init 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.736+0000 7fbbad4f8640 10 monclient: init 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.737+0000 7fbbad4f8640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.737+0000 7fbbad4f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.737+0000 7fbbad4f8640 10 monclient(hunting): picked mon.noname-c con 0x7fbba806c250 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.737+0000 7fbbad4f8640 10 monclient(hunting): picked mon.noname-b con 0x7fbba806c630 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.737+0000 7fbbad4f8640 10 monclient(hunting): picked mon.noname-a con 0x7fbba8072400 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.737+0000 7fbbad4f8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba80ae670 con 0x7fbba8072400 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.737+0000 7fbbad4f8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba813deb0 con 0x7fbba806c630 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.737+0000 7fbbad4f8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba80af1a0 con 0x7fbba806c250 2024-03-29T10:46:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.737+0000 7fbbad4f8640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:23.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.737+0000 7fbbad4f8640 10 monclient(hunting): authenticate will time out at 1248.978611s 2024-03-29T10:46:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.740+0000 7fbba6ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fbba806c250 0x7fbba813a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:51112/0 (socket says 172.21.15.114:51112) 2024-03-29T10:46:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.740+0000 7fbba6ffd640 1 -- 172.21.15.114:0/1108162843 learned_addr learned my addr 172.21.15.114:0/1108162843 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 10 client.?.objecter ms_handle_connect 0x7fbba806c250 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552014817 0 0) 0x7fbba80af1a0 con 0x7fbba806c250 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb88004eb0 con 0x7fbba806c250 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3899985907 0 0) 0x7fbb88004eb0 con 0x7fbba806c250 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbba80af1a0 con 0x7fbba806c250 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fbb98001630 con 0x7fbba806c250 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:23.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 10 monclient(hunting): dump: 2024-03-29T10:46:23.728 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:23.728 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:23.728 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:23.728 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:23.728 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:23.728 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:23.728 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:23.728 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:23.728 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:23.728 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:23.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.741+0000 7fbb9ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:23.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.742+0000 7fbb9ffff640 10 client.?.objecter ms_handle_connect 0x7fbba8072400 2024-03-29T10:46:23.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.742+0000 7fbb9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:23.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.742+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990078052 0 0) 0x7fbba80ae670 con 0x7fbba8072400 2024-03-29T10:46:23.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.742+0000 7fbb9ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:23.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.742+0000 7fbb9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:23.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.742+0000 7fbb9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:23.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.742+0000 7fbb9ffff640 10 monclient(hunting): my global_id is 4488 2024-03-29T10:46:23.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.742+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb88004eb0 con 0x7fbba8072400 2024-03-29T10:46:23.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.742+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1231905221 0 0) 0x7fbba80af1a0 con 0x7fbba806c250 2024-03-29T10:46:23.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.742+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 >> v1:172.21.15.176:6789/0 conn(0x7fbba806c630 legacy=0x7fbba80aea90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:23.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.746+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 >> v1:172.21.15.114:6789/0 conn(0x7fbba8072400 legacy=0x7fbba81ffc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:23.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbb9ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:23.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbb9ffff640 1 monclient: found mon.c 2024-03-29T10:46:23.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbb9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:23.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbb9ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:23.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba80af330 con 0x7fbba806c250 2024-03-29T10:46:23.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbb9ffff640 10 monclient: _finish_auth 0 2024-03-29T10:46:23.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbb9ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:23.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbb9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:23.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbbad4f8640 5 monclient: authenticate success, global_id 4544 2024-03-29T10:46:23.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbbad4f8640 10 monclient: _renew_subs 2024-03-29T10:46:23.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbbad4f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:23.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1108162843 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba813d990 con 0x7fbba806c250 2024-03-29T10:46:23.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbbad4f8640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:23.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbbad4f8640 10 monclient: _renew_subs 2024-03-29T10:46:23.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbbad4f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:23.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1108162843 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba813e5b0 con 0x7fbba806c250 2024-03-29T10:46:23.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.747+0000 7fbbad4f8640 1 librados: init done 2024-03-29T10:46:23.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.749+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb98004c70 con 0x7fbba806c250 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.749+0000 7fbb9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.749+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fbb98004f80 con 0x7fbba806c250 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.749+0000 7fbb9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.749+0000 7fbb9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.749+0000 7fbb9ffff640 10 monclient: dump: 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:23.734 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:23.735 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:23.735 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:23.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.750+0000 7fbba57fa640 4 set_mon_vals no callback set 2024-03-29T10:46:23.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.750+0000 7fbbad4f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:23.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.751+0000 7fbbad4f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:23.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.751+0000 7fbbad4f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:23.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.751+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1108162843 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8c005080 con 0x7fbba806c250 2024-03-29T10:46:23.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.751+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fbb9801af50 con 0x7fbba806c250 2024-03-29T10:46:23.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.752+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fbb98084da0 con 0x7fbba806c250 2024-03-29T10:46:23.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.752+0000 7fbb9ffff640 10 client.4544.objecter ms_dispatch 0x7fbba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:23.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.752+0000 7fbb9ffff640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:23.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.752+0000 7fbb9ffff640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:23.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.753+0000 7fbb9ffff640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T10:46:23.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.753+0000 7fbb9e7fc640 10 monclient: get_auth_request con 0x7fbb8806dad0 auth_method 0 2024-03-29T10:46:23.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.753+0000 7fbb9ffff640 10 client.4544.objecter ms_handle_connect 0x7fbb8806dad0 2024-03-29T10:46:23.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.760+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb98050680 con 0x7fbba806c250 2024-03-29T10:46:23.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.760+0000 7fbb9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:23.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:23.760+0000 7fbb9ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:24.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.082+0000 7fbbad4f8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:24.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.082+0000 7fbbad4f8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:24.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.082+0000 7fbbad4f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:24.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.082+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1108162843 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb8c005330 con 0x7fbba806c250 2024-03-29T10:46:24.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.083+0000 7fbb9ffff640 1 -- 172.21.15.114:0/1108162843 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fbb980525b0 con 0x7fbba806c250 2024-03-29T10:46:24.067 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:46:24.067 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:24.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.083+0000 7fbb9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:24.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.083+0000 7fbb9ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:24.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.087+0000 7fbbad4f8640 10 librados: watch_flush enter 2024-03-29T10:46:24.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.087+0000 7fbbad4f8640 10 librados: watch_flush exit 2024-03-29T10:46:24.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.087+0000 7fbbad4f8640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T10:46:24.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.087+0000 7fbbad4f8640 10 client.4544.objecter successfully canceled tick 2024-03-29T10:46:24.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.088+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1108162843 >> v1:172.21.15.114:6816/775044307 conn(0x7fbb8806dad0 legacy=0x7fbb8806ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:24.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.088+0000 7fbbad4f8640 10 monclient: shutdown 2024-03-29T10:46:24.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.088+0000 7fbbad4f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:24.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.088+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1108162843 >> v1:172.21.15.114:6790/0 conn(0x7fbba806c250 legacy=0x7fbba813a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:24.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.088+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1108162843 shutdown_connections 2024-03-29T10:46:24.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.088+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1108162843 >> 172.21.15.114:0/1108162843 conn(0x7fbba8067610 msgr2=0x7fbba81436d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:24.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.088+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1108162843 shutdown_connections 2024-03-29T10:46:24.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.088+0000 7fbbad4f8640 1 -- 172.21.15.114:0/1108162843 wait complete. 2024-03-29T10:46:24.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:24.089+0000 7fbbad4f8640 1 librados: shutdown 2024-03-29T10:46:24.089 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:30.894 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:31.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.054+0000 7f9c707ef640 10 monclient: get_monmap_and_config 2024-03-29T10:46:31.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.054+0000 7f9c707ef640 10 monclient: build_initial_monmap 2024-03-29T10:46:31.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.054+0000 7f9c707ef640 10 monclient: monmap: 2024-03-29T10:46:31.038 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:31.038 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:31.038 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:46:31.056146+0000 2024-03-29T10:46:31.039 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:46:31.056146+0000 2024-03-29T10:46:31.039 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:31.039 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:31.039 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:31.039 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:46:31.039 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:46:31.039 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:31.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.055+0000 7f9c707ef640 1 Processor -- start 2024-03-29T10:46:31.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.056+0000 7f9c707ef640 1 -- start start 2024-03-29T10:46:31.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.056+0000 7f9c707ef640 10 monclient: init 2024-03-29T10:46:31.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.058+0000 7f9c707ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:31.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.058+0000 7f9c707ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:31.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.058+0000 7f9c707ef640 10 monclient(hunting): picked mon.noname-a con 0x7f9c68071de0 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:31.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.058+0000 7f9c707ef640 10 monclient(hunting): picked mon.noname-b con 0x7f9c680721c0 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:31.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.058+0000 7f9c707ef640 10 monclient(hunting): picked mon.noname-c con 0x7f9c68145ae0 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:31.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.059+0000 7f9c707ef640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c68145650 con 0x7f9c68071de0 2024-03-29T10:46:31.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.059+0000 7f9c707ef640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c680ae260 con 0x7f9c68145ae0 2024-03-29T10:46:31.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.059+0000 7f9c707ef640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c681420d0 con 0x7f9c680721c0 2024-03-29T10:46:31.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.059+0000 7f9c707ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:31.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.059+0000 7f9c707ef640 10 monclient(hunting): authenticate will time out at 1256.300630s 2024-03-29T10:46:31.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.059+0000 7f9c6dd8a640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f9c680721c0 0x7f9c6813f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55140/0 (socket says 172.21.15.114:55140) 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.059+0000 7f9c6dd8a640 1 -- 172.21.15.114:0/3433097941 learned_addr learned my addr 172.21.15.114:0/3433097941 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 1 -- 172.21.15.114:0/3433097941 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725534667 0 0) 0x7f9c681420d0 con 0x7f9c680721c0 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 10 monclient(hunting): _negotiate 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 1 -- 172.21.15.114:0/3433097941 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c54003520 con 0x7f9c680721c0 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 1 -- 172.21.15.114:0/3433097941 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925341073 0 0) 0x7f9c68145650 con 0x7f9c68071de0 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 10 monclient(hunting): _negotiate 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 10 monclient(hunting): my global_id is 4491 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.060+0000 7f9c6d589640 1 -- 172.21.15.114:0/3433097941 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c681420d0 con 0x7f9c68071de0 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 1 -- 172.21.15.114:0/3433097941 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 328027156 0 0) 0x7f9c54003520 con 0x7f9c680721c0 2024-03-29T10:46:31.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 1 -- 172.21.15.114:0/3433097941 >> v1:172.21.15.176:6789/0 conn(0x7f9c68145ae0 legacy=0x7f9c68145f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:31.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 1 -- 172.21.15.114:0/3433097941 >> v1:172.21.15.114:6789/0 conn(0x7f9c68071de0 legacy=0x7f9c680aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:31.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 10 monclient: _finish_hunting 0 2024-03-29T10:46:31.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 1 monclient: found mon.noname-b 2024-03-29T10:46:31.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:31.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:46:31.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 1 -- 172.21.15.114:0/3433097941 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c680b0590 con 0x7f9c680721c0 2024-03-29T10:46:31.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 10 monclient: _finish_auth 0 2024-03-29T10:46:31.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 10 monclient: _check_auth_tickets 2024-03-29T10:46:31.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:31.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c707ef640 5 monclient: authenticate success, global_id 4547 2024-03-29T10:46:31.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c707ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 1 -- 172.21.15.114:0/3433097941 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9c580015c0 con 0x7f9c680721c0 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c6d589640 10 monclient: dump: 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:31.049 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.061+0000 7f9c707ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.062+0000 7f9c6d589640 1 -- 172.21.15.114:0/3433097941 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c580017e0 con 0x7f9c680721c0 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.062+0000 7f9c6d589640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.062+0000 7f9c6d589640 1 -- 172.21.15.114:0/3433097941 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9c58001af0 con 0x7f9c680721c0 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.062+0000 7f9c707ef640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.062+0000 7f9c707ef640 4 set_mon_vals no callback set 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.062+0000 7f9c6d589640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.062+0000 7f9c6d589640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.062+0000 7f9c6d589640 10 monclient: dump: 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:31.050 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.063+0000 7f9c707ef640 10 monclient: shutdown 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.063+0000 7f9c707ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.063+0000 7f9c707ef640 1 -- 172.21.15.114:0/3433097941 >> v1:172.21.15.114:6790/0 conn(0x7f9c680721c0 legacy=0x7f9c6813f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.063+0000 7f9c707ef640 1 -- 172.21.15.114:0/3433097941 shutdown_connections 2024-03-29T10:46:31.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.063+0000 7f9c707ef640 1 -- 172.21.15.114:0/3433097941 >> 172.21.15.114:0/3433097941 conn(0x7f9c6806b660 msgr2=0x7f9c6806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:31.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.063+0000 7f9c707ef640 1 -- 172.21.15.114:0/3433097941 shutdown_connections 2024-03-29T10:46:31.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.063+0000 7f9c707ef640 1 -- 172.21.15.114:0/3433097941 wait complete. 2024-03-29T10:46:31.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.064+0000 7f9c707ef640 10 monclient: build_initial_monmap 2024-03-29T10:46:31.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.064+0000 7f9c707ef640 10 monclient: monmap: 2024-03-29T10:46:31.058 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:31.058 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:31.059 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:46:31.059 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:46:31.059 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:31.059 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:31.059 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:31.059 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:46:31.059 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:46:31.059 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:31.059 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.065+0000 7f9c707ef640 1 librados: starting msgr at 2024-03-29T10:46:31.059 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.065+0000 7f9c707ef640 1 librados: starting objecter 2024-03-29T10:46:31.059 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.065+0000 7f9c707ef640 1 Processor -- start 2024-03-29T10:46:31.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.065+0000 7f9c707ef640 1 -- start start 2024-03-29T10:46:31.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.065+0000 7f9c707ef640 1 librados: setting wanted keys 2024-03-29T10:46:31.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.065+0000 7f9c707ef640 1 librados: calling monclient init 2024-03-29T10:46:31.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.065+0000 7f9c707ef640 10 monclient: init 2024-03-29T10:46:31.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c707ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:31.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c707ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:31.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c707ef640 10 monclient(hunting): picked mon.noname-a con 0x7f9c68071de0 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:31.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c707ef640 10 monclient(hunting): picked mon.noname-b con 0x7f9c68145ae0 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:31.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c707ef640 10 monclient(hunting): picked mon.noname-c con 0x7f9c681df8d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:31.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c707ef640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c680706b0 con 0x7f9c68071de0 2024-03-29T10:46:31.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c707ef640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c681deca0 con 0x7f9c68145ae0 2024-03-29T10:46:31.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c707ef640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6806f420 con 0x7f9c681df8d0 2024-03-29T10:46:31.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c707ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:31.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c707ef640 10 monclient(hunting): authenticate will time out at 1256.307617s 2024-03-29T10:46:31.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c6ed8c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f9c681df8d0 0x7f9c680710a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34658/0 (socket says 172.21.15.114:34658) 2024-03-29T10:46:31.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.066+0000 7f9c6ed8c640 1 -- 172.21.15.114:0/1699878469 learned_addr learned my addr 172.21.15.114:0/1699878469 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:31.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 client.?.objecter ms_handle_connect 0x7f9c681df8d0 2024-03-29T10:46:31.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:31.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 client.?.objecter ms_handle_connect 0x7f9c68071de0 2024-03-29T10:46:31.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:31.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094716826 0 0) 0x7f9c6806f420 con 0x7f9c681df8d0 2024-03-29T10:46:31.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:31.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:31.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:31.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:46:31.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c4c004eb0 con 0x7f9c681df8d0 2024-03-29T10:46:31.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553824706 0 0) 0x7f9c680706b0 con 0x7f9c68071de0 2024-03-29T10:46:31.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:31.063 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:31.064 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:31.064 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.067+0000 7f9c5effd640 10 monclient(hunting): my global_id is 4494 2024-03-29T10:46:31.064 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c6806f420 con 0x7f9c68071de0 2024-03-29T10:46:31.064 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2190886314 0 0) 0x7f9c4c004eb0 con 0x7f9c681df8d0 2024-03-29T10:46:31.064 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c680706b0 con 0x7f9c681df8d0 2024-03-29T10:46:31.064 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9c600017f0 con 0x7f9c681df8d0 2024-03-29T10:46:31.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:31.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:31.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 10 monclient(hunting): dump: 2024-03-29T10:46:31.065 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:31.065 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:31.065 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:31.066 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:31.066 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:31.066 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:31.066 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:31.066 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:31.066 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:31.066 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:31.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:31.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1863069630 0 0) 0x7f9c6806f420 con 0x7f9c68071de0 2024-03-29T10:46:31.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c4c004eb0 con 0x7f9c68071de0 2024-03-29T10:46:31.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9c64001b50 con 0x7f9c68071de0 2024-03-29T10:46:31.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:31.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:31.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 10 monclient(hunting): dump: 2024-03-29T10:46:31.068 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:31.068 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:31.068 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:31.068 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:31.068 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:31.069 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:31.069 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:31.069 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:31.069 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:31.069 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:31.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4094902688 0 0) 0x7f9c680706b0 con 0x7f9c681df8d0 2024-03-29T10:46:31.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.068+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 >> v1:172.21.15.176:6789/0 conn(0x7f9c68145ae0 legacy=0x7f9c681df1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 >> v1:172.21.15.114:6789/0 conn(0x7f9c68071de0 legacy=0x7f9c6806ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c5effd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c5effd640 1 monclient: found mon.c 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c5effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:31.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c68070ad0 con 0x7f9c681df8d0 2024-03-29T10:46:31.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c5effd640 10 monclient: _finish_auth 0 2024-03-29T10:46:31.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c5effd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:31.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:31.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c707ef640 5 monclient: authenticate success, global_id 4550 2024-03-29T10:46:31.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c707ef640 10 monclient: _renew_subs 2024-03-29T10:46:31.071 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c707ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:31.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c707ef640 1 -- 172.21.15.114:0/1699878469 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6806e250 con 0x7f9c681df8d0 2024-03-29T10:46:31.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c707ef640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:31.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c707ef640 10 monclient: _renew_subs 2024-03-29T10:46:31.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c707ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:31.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c707ef640 1 -- 172.21.15.114:0/1699878469 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c68071990 con 0x7f9c681df8d0 2024-03-29T10:46:31.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c707ef640 1 librados: init done 2024-03-29T10:46:31.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c60001950 con 0x7f9c681df8d0 2024-03-29T10:46:31.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:31.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.069+0000 7f9c6cd88640 4 set_mon_vals no callback set 2024-03-29T10:46:31.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.070+0000 7f9c707ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:31.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.070+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9c60004aa0 con 0x7f9c681df8d0 2024-03-29T10:46:31.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.070+0000 7f9c5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:31.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.070+0000 7f9c5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:31.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.070+0000 7f9c5effd640 10 monclient: dump: 2024-03-29T10:46:31.074 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:31.074 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:31.074 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:31.074 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:31.075 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:31.075 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:31.075 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:31.075 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:31.075 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:31.075 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:31.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.070+0000 7f9c707ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:31.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.070+0000 7f9c707ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:31.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.070+0000 7f9c707ef640 1 -- 172.21.15.114:0/1699878469 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c50005080 con 0x7f9c681df8d0 2024-03-29T10:46:31.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.071+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f9c6001aa70 con 0x7f9c681df8d0 2024-03-29T10:46:31.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.072+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f9c600849e0 con 0x7f9c681df8d0 2024-03-29T10:46:31.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.072+0000 7f9c5effd640 10 client.4550.objecter ms_dispatch 0x7f9c68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:31.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.072+0000 7f9c5effd640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:31.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.072+0000 7f9c5effd640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:31.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.072+0000 7f9c5effd640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T10:46:31.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.072+0000 7f9c6e58b640 10 monclient: get_auth_request con 0x7f9c4c06de70 auth_method 0 2024-03-29T10:46:31.077 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.073+0000 7f9c5effd640 10 client.4550.objecter ms_handle_connect 0x7f9c4c06de70 2024-03-29T10:46:31.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.080+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c600502c0 con 0x7f9c681df8d0 2024-03-29T10:46:31.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.081+0000 7f9c5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:31.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.081+0000 7f9c5effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:31.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.415+0000 7f9c707ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:31.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.415+0000 7f9c707ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:31.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.415+0000 7f9c707ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:46:31.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.415+0000 7f9c707ef640 1 -- 172.21.15.114:0/1699878469 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c50005330 con 0x7f9c681df8d0 2024-03-29T10:46:31.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.419+0000 7f9c5effd640 1 -- 172.21.15.114:0/1699878469 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f9c600521f0 con 0x7f9c681df8d0 2024-03-29T10:46:31.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.419+0000 7f9c5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:31.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.419+0000 7f9c5effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:31.403 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:46:31.403 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:31.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.423+0000 7f9c707ef640 10 librados: watch_flush enter 2024-03-29T10:46:31.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.423+0000 7f9c707ef640 10 librados: watch_flush exit 2024-03-29T10:46:31.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.423+0000 7f9c707ef640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T10:46:31.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.423+0000 7f9c707ef640 10 client.4550.objecter successfully canceled tick 2024-03-29T10:46:31.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.423+0000 7f9c707ef640 1 -- 172.21.15.114:0/1699878469 >> v1:172.21.15.114:6816/775044307 conn(0x7f9c4c06de70 legacy=0x7f9c4c070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:31.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.423+0000 7f9c707ef640 10 monclient: shutdown 2024-03-29T10:46:31.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.423+0000 7f9c707ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:31.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.423+0000 7f9c707ef640 1 -- 172.21.15.114:0/1699878469 >> v1:172.21.15.114:6790/0 conn(0x7f9c681df8d0 legacy=0x7f9c680710a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:31.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.425+0000 7f9c707ef640 1 -- 172.21.15.114:0/1699878469 shutdown_connections 2024-03-29T10:46:31.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.425+0000 7f9c707ef640 1 -- 172.21.15.114:0/1699878469 >> 172.21.15.114:0/1699878469 conn(0x7f9c6806b660 msgr2=0x7f9c680a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:31.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.427+0000 7f9c707ef640 1 -- 172.21.15.114:0/1699878469 shutdown_connections 2024-03-29T10:46:31.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.427+0000 7f9c707ef640 1 -- 172.21.15.114:0/1699878469 wait complete. 2024-03-29T10:46:31.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:31.427+0000 7f9c707ef640 1 librados: shutdown 2024-03-29T10:46:31.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:38.237 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:38.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.408+0000 7f492b59e640 10 monclient: get_monmap_and_config 2024-03-29T10:46:38.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.408+0000 7f492b59e640 10 monclient: build_initial_monmap 2024-03-29T10:46:38.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.408+0000 7f492b59e640 10 monclient: monmap: 2024-03-29T10:46:38.391 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:38.391 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:38.391 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:46:38.409396+0000 2024-03-29T10:46:38.391 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:46:38.409396+0000 2024-03-29T10:46:38.391 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:38.391 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:38.392 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:38.392 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:46:38.392 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:46:38.392 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:38.392 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.408+0000 7f492b59e640 1 Processor -- start 2024-03-29T10:46:38.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.409+0000 7f492b59e640 1 -- start start 2024-03-29T10:46:38.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.409+0000 7f492b59e640 10 monclient: init 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492b59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f492c0b2800 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f492c142b20 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f492c13c040 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492b59e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c0aad10 con 0x7f492c13c040 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492b59e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c141f30 con 0x7f492c142b20 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492b59e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c0aaf60 con 0x7f492c0b2800 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492b59e640 10 monclient(hunting): authenticate will time out at 1263.651496s 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492a59c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f492c0b2800 0x7f492c142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:59246/0 (socket says 172.21.15.114:59246) 2024-03-29T10:46:38.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.410+0000 7f492a59c640 1 -- 172.21.15.114:0/2069516243 learned_addr learned my addr 172.21.15.114:0/2069516243 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:38.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 1 -- 172.21.15.114:0/2069516243 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948741809 0 0) 0x7f492c0aaf60 con 0x7f492c0b2800 2024-03-29T10:46:38.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient(hunting): _negotiate 2024-03-29T10:46:38.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:38.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:38.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:46:38.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 1 -- 172.21.15.114:0/2069516243 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4914003520 con 0x7f492c0b2800 2024-03-29T10:46:38.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 1 -- 172.21.15.114:0/2069516243 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709616102 0 0) 0x7f492c0aad10 con 0x7f492c13c040 2024-03-29T10:46:38.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient(hunting): _negotiate 2024-03-29T10:46:38.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:38.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:38.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient(hunting): my global_id is 4497 2024-03-29T10:46:38.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 1 -- 172.21.15.114:0/2069516243 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492c0aaf60 con 0x7f492c13c040 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 1 -- 172.21.15.114:0/2069516243 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2627923850 0 0) 0x7f4914003520 con 0x7f492c0b2800 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 1 -- 172.21.15.114:0/2069516243 >> v1:172.21.15.176:6789/0 conn(0x7f492c142b20 legacy=0x7f492c13b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 1 -- 172.21.15.114:0/2069516243 >> v1:172.21.15.114:6789/0 conn(0x7f492c13c040 legacy=0x7f492c14d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient: _finish_hunting 0 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 1 monclient: found mon.noname-b 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 1 -- 172.21.15.114:0/2069516243 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c0b3900 con 0x7f492c0b2800 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient: _finish_auth 0 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient: _check_auth_tickets 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:38.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 1 -- 172.21.15.114:0/2069516243 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4920001690 con 0x7f492c0b2800 2024-03-29T10:46:38.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:38.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:38.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492959a640 10 monclient: dump: 2024-03-29T10:46:38.404 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:38.404 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:38.404 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:38.404 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:38.404 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:38.404 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:38.404 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492b59e640 5 monclient: authenticate success, global_id 4553 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.411+0000 7f492b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.412+0000 7f492959a640 1 -- 172.21.15.114:0/2069516243 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4920001870 con 0x7f492c0b2800 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.412+0000 7f492959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.412+0000 7f492959a640 1 -- 172.21.15.114:0/2069516243 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4920001c80 con 0x7f492c0b2800 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.412+0000 7f492959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.412+0000 7f492959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.412+0000 7f492959a640 10 monclient: dump: 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:38.405 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.412+0000 7f492b59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.412+0000 7f492b59e640 4 set_mon_vals no callback set 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.412+0000 7f492b59e640 10 monclient: shutdown 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.412+0000 7f492b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.412+0000 7f492b59e640 1 -- 172.21.15.114:0/2069516243 >> v1:172.21.15.114:6790/0 conn(0x7f492c0b2800 legacy=0x7f492c142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.413+0000 7f492b59e640 1 -- 172.21.15.114:0/2069516243 shutdown_connections 2024-03-29T10:46:38.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.413+0000 7f492b59e640 1 -- 172.21.15.114:0/2069516243 >> 172.21.15.114:0/2069516243 conn(0x7f492c06d0e0 msgr2=0x7f492c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.413+0000 7f492b59e640 1 -- 172.21.15.114:0/2069516243 shutdown_connections 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.413+0000 7f492b59e640 1 -- 172.21.15.114:0/2069516243 wait complete. 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.414+0000 7f492b59e640 10 monclient: build_initial_monmap 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.414+0000 7f492b59e640 10 monclient: monmap: 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:46:38.407 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.414+0000 7f492b59e640 1 librados: starting msgr at 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.414+0000 7f492b59e640 1 librados: starting objecter 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.414+0000 7f492b59e640 1 Processor -- start 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.414+0000 7f492b59e640 1 -- start start 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.414+0000 7f492b59e640 1 librados: setting wanted keys 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.414+0000 7f492b59e640 1 librados: calling monclient init 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.414+0000 7f492b59e640 10 monclient: init 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492b59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f492c0b2800 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492a59c640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f492c0b2800 0x7f492c1da5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48650/0 (socket says 172.21.15.114:48650) 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492a59c640 1 -- 172.21.15.114:0/2819761376 learned_addr learned my addr 172.21.15.114:0/2819761376 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:38.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f492c13c040 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f492c142b20 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c145a90 con 0x7f492c0b2800 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c137700 con 0x7f492c142b20 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c0ac0a0 con 0x7f492c13c040 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.415+0000 7f492b59e640 10 monclient(hunting): authenticate will time out at 1263.656936s 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 client.?.objecter ms_handle_connect 0x7f492c0b2800 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 client.?.objecter ms_handle_connect 0x7f492c13c040 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205242689 0 0) 0x7f492c145a90 con 0x7f492c0b2800 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:38.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 monclient(hunting): my global_id is 4500 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f490c004eb0 con 0x7f492c0b2800 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885248455 0 0) 0x7f492c0ac0a0 con 0x7f492c13c040 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 monclient(hunting): my global_id is 4556 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492c145a90 con 0x7f492c13c040 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 698572915 0 0) 0x7f490c004eb0 con 0x7f492c0b2800 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f492c0ac0a0 con 0x7f492c0b2800 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4920001bc0 con 0x7f492c0b2800 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.416+0000 7f491affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:38.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 10 monclient(hunting): dump: 2024-03-29T10:46:38.411 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:38.411 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:38.411 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:38.411 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:38.411 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:38.411 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:38.411 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:38.411 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:38.411 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 68561037 0 0) 0x7f492c145a90 con 0x7f492c13c040 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f490c004eb0 con 0x7f492c13c040 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f491c001850 con 0x7f492c13c040 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 10 monclient(hunting): dump: 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2780682041 0 0) 0x7f492c0ac0a0 con 0x7f492c0b2800 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 >> v1:172.21.15.114:6790/0 conn(0x7f492c13c040 legacy=0x7f492c1368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 >> v1:172.21.15.176:6789/0 conn(0x7f492c142b20 legacy=0x7f492c136ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 1 monclient: found mon.a 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c145f20 con 0x7f492c0b2800 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 10 monclient: _finish_auth 0 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f491affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f492b59e640 5 monclient: authenticate success, global_id 4500 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f492b59e640 10 monclient: _renew_subs 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f492b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c145570 con 0x7f492c0b2800 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f492b59e640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f492b59e640 10 monclient: _renew_subs 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f492b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c1daec0 con 0x7f492c0b2800 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.417+0000 7f492b59e640 1 librados: init done 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.418+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4920004bf0 con 0x7f492c0b2800 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.418+0000 7f491affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.418+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4920005000 con 0x7f492c0b2800 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.418+0000 7f491affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.418+0000 7f491affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.418+0000 7f491affd640 10 monclient: dump: 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:38.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.418+0000 7f4928d99640 4 set_mon_vals no callback set 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.421+0000 7f492b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.421+0000 7f492b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.421+0000 7f492b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.421+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4910005080 con 0x7f492c0b2800 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.421+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f4920005160 con 0x7f492c0b2800 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.422+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f4920084d30 con 0x7f492c0b2800 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.422+0000 7f491affd640 10 client.4500.objecter ms_dispatch 0x7f492c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.422+0000 7f491affd640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.422+0000 7f491affd640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.422+0000 7f491affd640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.422+0000 7f4929d9b640 10 monclient: get_auth_request con 0x7f490c06de90 auth_method 0 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.422+0000 7f491affd640 10 client.4500.objecter ms_handle_connect 0x7f490c06de90 2024-03-29T10:46:38.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.429+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4920050610 con 0x7f492c0b2800 2024-03-29T10:46:38.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.429+0000 7f491affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:38.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.429+0000 7f491affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:38.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.797+0000 7f492b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:38.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.797+0000 7f492b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:38.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.797+0000 7f492b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:46:38.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.797+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4910005330 con 0x7f492c0b2800 2024-03-29T10:46:38.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.798+0000 7f491affd640 1 -- 172.21.15.114:0/2819761376 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f4920052540 con 0x7f492c0b2800 2024-03-29T10:46:38.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.798+0000 7f491affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:38.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.798+0000 7f491affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:38.781 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:46:38.782 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:38.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.802+0000 7f492b59e640 10 librados: watch_flush enter 2024-03-29T10:46:38.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.802+0000 7f492b59e640 10 librados: watch_flush exit 2024-03-29T10:46:38.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.802+0000 7f492b59e640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T10:46:38.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.802+0000 7f492b59e640 10 client.4500.objecter successfully canceled tick 2024-03-29T10:46:38.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.802+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 >> v1:172.21.15.114:6816/775044307 conn(0x7f490c06de90 legacy=0x7f490c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.802+0000 7f492b59e640 10 monclient: shutdown 2024-03-29T10:46:38.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.802+0000 7f492b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:38.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.802+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 >> v1:172.21.15.114:6789/0 conn(0x7f492c0b2800 legacy=0x7f492c1da5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.802+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 shutdown_connections 2024-03-29T10:46:38.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.802+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 >> 172.21.15.114:0/2819761376 conn(0x7f492c06d0e0 msgr2=0x7f492c139f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:38.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.803+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 shutdown_connections 2024-03-29T10:46:38.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.803+0000 7f492b59e640 1 -- 172.21.15.114:0/2819761376 wait complete. 2024-03-29T10:46:38.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:38.803+0000 7f492b59e640 1 librados: shutdown 2024-03-29T10:46:38.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:45.606 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:45.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.767+0000 7fae904b3640 10 monclient: get_monmap_and_config 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.767+0000 7fae904b3640 10 monclient: build_initial_monmap 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.767+0000 7fae904b3640 10 monclient: monmap: 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:46:45.768215+0000 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:46:45.768215+0000 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:46:45.751 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:45.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.768+0000 7fae904b3640 1 Processor -- start 2024-03-29T10:46:45.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.769+0000 7fae904b3640 1 -- start start 2024-03-29T10:46:45.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.769+0000 7fae904b3640 10 monclient: init 2024-03-29T10:46:45.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.770+0000 7fae904b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:45.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.770+0000 7fae904b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:45.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.770+0000 7fae904b3640 10 monclient(hunting): picked mon.noname-a con 0x7fae880ae250 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:45.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.770+0000 7fae904b3640 10 monclient(hunting): picked mon.noname-c con 0x7fae8813f970 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:45.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.770+0000 7fae904b3640 10 monclient(hunting): picked mon.noname-b con 0x7fae88142800 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:45.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.770+0000 7fae904b3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae880afe00 con 0x7fae880ae250 2024-03-29T10:46:45.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.770+0000 7fae904b3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae88142c10 con 0x7fae8813f970 2024-03-29T10:46:45.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.770+0000 7fae904b3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae880afb20 con 0x7fae88142800 2024-03-29T10:46:45.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.770+0000 7fae904b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:45.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.770+0000 7fae904b3640 10 monclient(hunting): authenticate will time out at 1271.010504s 2024-03-29T10:46:45.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.774+0000 7fae8da4e640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fae8813f970 0x7fae881420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:38367/0 (socket says 172.21.15.114:38367) 2024-03-29T10:46:45.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.774+0000 7fae8da4e640 1 -- 172.21.15.114:0/3306968917 learned_addr learned my addr 172.21.15.114:0/3306968917 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:45.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.775+0000 7fae8d24d640 1 -- 172.21.15.114:0/3306968917 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765177376 0 0) 0x7fae88142c10 con 0x7fae8813f970 2024-03-29T10:46:45.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.775+0000 7fae8d24d640 10 monclient(hunting): _negotiate 2024-03-29T10:46:45.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.775+0000 7fae8d24d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:45.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.775+0000 7fae8d24d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:45.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.775+0000 7fae8d24d640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:46:45.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.775+0000 7fae8d24d640 1 -- 172.21.15.114:0/3306968917 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae74003520 con 0x7fae8813f970 2024-03-29T10:46:45.760 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.777+0000 7fae8d24d640 1 -- 172.21.15.114:0/3306968917 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849459918 0 0) 0x7fae880afb20 con 0x7fae88142800 2024-03-29T10:46:45.760 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.777+0000 7fae8d24d640 10 monclient(hunting): _negotiate 2024-03-29T10:46:45.760 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.777+0000 7fae8d24d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:45.760 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.777+0000 7fae8d24d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:45.760 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.777+0000 7fae8d24d640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:46:45.760 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.777+0000 7fae8d24d640 1 -- 172.21.15.114:0/3306968917 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae88142c10 con 0x7fae88142800 2024-03-29T10:46:45.760 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.778+0000 7fae8d24d640 1 -- 172.21.15.114:0/3306968917 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 507177556 0 0) 0x7fae88142c10 con 0x7fae88142800 2024-03-29T10:46:45.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.778+0000 7fae8d24d640 1 -- 172.21.15.114:0/3306968917 >> v1:172.21.15.176:6789/0 conn(0x7fae8813f970 legacy=0x7fae881420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:45.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.778+0000 7fae8d24d640 1 -- 172.21.15.114:0/3306968917 >> v1:172.21.15.114:6789/0 conn(0x7fae880ae250 legacy=0x7fae8813f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 10 monclient: _finish_hunting 0 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 1 monclient: found mon.noname-b 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 1 -- 172.21.15.114:0/3306968917 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae880aab80 con 0x7fae88142800 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 10 monclient: _finish_auth 0 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 10 monclient: _check_auth_tickets 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 1 -- 172.21.15.114:0/3306968917 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fae800019f0 con 0x7fae88142800 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae8d24d640 10 monclient: dump: 2024-03-29T10:46:45.765 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae904b3640 5 monclient: authenticate success, global_id 4559 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.780+0000 7fae904b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.781+0000 7fae8d24d640 1 -- 172.21.15.114:0/3306968917 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae80001b50 con 0x7fae88142800 2024-03-29T10:46:45.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.781+0000 7fae8d24d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.781+0000 7fae8d24d640 1 -- 172.21.15.114:0/3306968917 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fae80001fa0 con 0x7fae88142800 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.781+0000 7fae8d24d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.781+0000 7fae8d24d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.781+0000 7fae8d24d640 10 monclient: dump: 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.781+0000 7fae904b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:45.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.781+0000 7fae904b3640 4 set_mon_vals no callback set 2024-03-29T10:46:45.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.781+0000 7fae904b3640 10 monclient: shutdown 2024-03-29T10:46:45.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.781+0000 7fae904b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:45.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.782+0000 7fae904b3640 1 -- 172.21.15.114:0/3306968917 >> v1:172.21.15.114:6790/0 conn(0x7fae88142800 legacy=0x7fae8814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:45.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.782+0000 7fae904b3640 1 -- 172.21.15.114:0/3306968917 shutdown_connections 2024-03-29T10:46:45.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.782+0000 7fae904b3640 1 -- 172.21.15.114:0/3306968917 >> 172.21.15.114:0/3306968917 conn(0x7fae880b35e0 msgr2=0x7fae880b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:45.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.783+0000 7fae904b3640 1 -- 172.21.15.114:0/3306968917 shutdown_connections 2024-03-29T10:46:45.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.783+0000 7fae904b3640 1 -- 172.21.15.114:0/3306968917 wait complete. 2024-03-29T10:46:45.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.784+0000 7fae904b3640 10 monclient: build_initial_monmap 2024-03-29T10:46:45.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.784+0000 7fae904b3640 10 monclient: monmap: 2024-03-29T10:46:45.769 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:45.769 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:45.769 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.785+0000 7fae904b3640 1 librados: starting msgr at 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.785+0000 7fae904b3640 1 librados: starting objecter 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.785+0000 7fae904b3640 1 Processor -- start 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.785+0000 7fae904b3640 1 -- start start 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.785+0000 7fae904b3640 1 librados: setting wanted keys 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.785+0000 7fae904b3640 1 librados: calling monclient init 2024-03-29T10:46:45.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.786+0000 7fae904b3640 10 monclient: init 2024-03-29T10:46:45.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.787+0000 7fae904b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:45.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.787+0000 7fae904b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:45.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.787+0000 7fae904b3640 10 monclient(hunting): picked mon.noname-a con 0x7fae880ae250 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:45.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.789+0000 7fae904b3640 10 monclient(hunting): picked mon.noname-b con 0x7fae8813f970 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:45.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.789+0000 7fae904b3640 10 monclient(hunting): picked mon.noname-c con 0x7fae88142800 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:45.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.789+0000 7fae904b3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae88143c60 con 0x7fae880ae250 2024-03-29T10:46:45.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.789+0000 7fae904b3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae881d79c0 con 0x7fae8813f970 2024-03-29T10:46:45.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.789+0000 7fae904b3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae880ac390 con 0x7fae88142800 2024-03-29T10:46:45.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.789+0000 7fae904b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:45.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.789+0000 7fae904b3640 10 monclient(hunting): authenticate will time out at 1271.029197s 2024-03-29T10:46:45.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.796+0000 7fae8da4e640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fae8813f970 0x7fae881da6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:38453/0 (socket says 172.21.15.114:38453) 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.796+0000 7fae8da4e640 1 -- 172.21.15.114:0/4190774028 learned_addr learned my addr 172.21.15.114:0/4190774028 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 client.?.objecter ms_handle_connect 0x7fae8813f970 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 client.?.objecter ms_handle_connect 0x7fae88142800 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179122992 0 0) 0x7fae881d79c0 con 0x7fae8813f970 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae6c004eb0 con 0x7fae8813f970 2024-03-29T10:46:45.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604199716 0 0) 0x7fae880ac390 con 0x7fae88142800 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae881d79c0 con 0x7fae88142800 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3232634585 0 0) 0x7fae881d79c0 con 0x7fae88142800 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.797+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae880ac390 con 0x7fae88142800 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.798+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fae80001e60 con 0x7fae88142800 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.798+0000 7fae7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.798+0000 7fae7effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.798+0000 7fae7effd640 10 monclient(hunting): dump: 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:45.787 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.798+0000 7fae7effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.798+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3747250751 0 0) 0x7fae6c004eb0 con 0x7fae8813f970 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.798+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae881d79c0 con 0x7fae8813f970 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.798+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fae78001950 con 0x7fae8813f970 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.798+0000 7fae7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.798+0000 7fae7effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.798+0000 7fae7effd640 10 monclient(hunting): dump: 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2970352935 0 0) 0x7fae881d79c0 con 0x7fae8813f970 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 >> v1:172.21.15.114:6790/0 conn(0x7fae88142800 legacy=0x7fae881e1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 >> v1:172.21.15.114:6789/0 conn(0x7fae880ae250 legacy=0x7fae881d72b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae7effd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae7effd640 1 monclient: found mon.b 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae7effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae880ac520 con 0x7fae8813f970 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae7effd640 10 monclient: _finish_auth 0 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae7effd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae904b3640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae904b3640 10 monclient: _renew_subs 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae904b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae904b3640 1 -- 172.21.15.114:0/4190774028 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae881dadc0 con 0x7fae8813f970 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae904b3640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae904b3640 10 monclient: _renew_subs 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae904b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae904b3640 1 -- 172.21.15.114:0/4190774028 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae881db2a0 con 0x7fae8813f970 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.800+0000 7fae904b3640 1 librados: init done 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.801+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae78001ab0 con 0x7fae8813f970 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.801+0000 7fae7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.801+0000 7fae8ca4c640 4 set_mon_vals no callback set 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.802+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fae78001c90 con 0x7fae8813f970 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.802+0000 7fae7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.802+0000 7fae7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.802+0000 7fae7effd640 10 monclient: dump: 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:45.791 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:45.792 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:45.792 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:45.792 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:45.792 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:45.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.802+0000 7fae904b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:45.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.802+0000 7fae904b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:45.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.802+0000 7fae904b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:45.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.802+0000 7fae904b3640 1 -- 172.21.15.114:0/4190774028 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae70005080 con 0x7fae8813f970 2024-03-29T10:46:45.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.804+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fae7801a900 con 0x7fae8813f970 2024-03-29T10:46:45.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.805+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fae78084890 con 0x7fae8813f970 2024-03-29T10:46:45.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.805+0000 7fae7effd640 10 client.4474.objecter ms_dispatch 0x7fae88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:45.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.805+0000 7fae7effd640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:45.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.805+0000 7fae7effd640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:45.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.805+0000 7fae7effd640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T10:46:45.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.805+0000 7fae8e24f640 10 monclient: get_auth_request con 0x7fae6c076700 auth_method 0 2024-03-29T10:46:45.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.808+0000 7fae7effd640 10 client.4474.objecter ms_handle_connect 0x7fae6c076700 2024-03-29T10:46:45.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.811+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae7804ffa0 con 0x7fae8813f970 2024-03-29T10:46:45.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.811+0000 7fae7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:45.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:45.811+0000 7fae7effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:46.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.134+0000 7fae904b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:46.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.134+0000 7fae904b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:46.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.134+0000 7fae904b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:46.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.134+0000 7fae904b3640 1 -- 172.21.15.114:0/4190774028 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae70005330 con 0x7fae8813f970 2024-03-29T10:46:46.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.136+0000 7fae7effd640 1 -- 172.21.15.114:0/4190774028 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fae780520a0 con 0x7fae8813f970 2024-03-29T10:46:46.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.136+0000 7fae7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:46.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.136+0000 7fae7effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:46.120 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:46:46.120 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:46.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.140+0000 7fae904b3640 10 librados: watch_flush enter 2024-03-29T10:46:46.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.140+0000 7fae904b3640 10 librados: watch_flush exit 2024-03-29T10:46:46.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.140+0000 7fae904b3640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T10:46:46.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.140+0000 7fae904b3640 10 client.4474.objecter successfully canceled tick 2024-03-29T10:46:46.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.140+0000 7fae904b3640 1 -- 172.21.15.114:0/4190774028 >> v1:172.21.15.114:6816/775044307 conn(0x7fae6c076700 legacy=0x7fae6c078bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:46.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.140+0000 7fae904b3640 10 monclient: shutdown 2024-03-29T10:46:46.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.140+0000 7fae904b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:46.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.141+0000 7fae904b3640 1 -- 172.21.15.114:0/4190774028 >> v1:172.21.15.176:6789/0 conn(0x7fae8813f970 legacy=0x7fae881da6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:46.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.141+0000 7fae904b3640 1 -- 172.21.15.114:0/4190774028 shutdown_connections 2024-03-29T10:46:46.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.141+0000 7fae904b3640 1 -- 172.21.15.114:0/4190774028 >> 172.21.15.114:0/4190774028 conn(0x7fae880b35e0 msgr2=0x7fae880b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:46.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.141+0000 7fae904b3640 1 -- 172.21.15.114:0/4190774028 shutdown_connections 2024-03-29T10:46:46.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.141+0000 7fae904b3640 1 -- 172.21.15.114:0/4190774028 wait complete. 2024-03-29T10:46:46.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:46.141+0000 7fae904b3640 1 librados: shutdown 2024-03-29T10:46:46.143 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:52.951 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:53.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.131+0000 7f27df515640 10 monclient: get_monmap_and_config 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.131+0000 7f27df515640 10 monclient: build_initial_monmap 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.131+0000 7f27df515640 10 monclient: monmap: 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:46:53.133141+0000 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:46:53.133141+0000 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:46:53.115 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:53.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.133+0000 7f27df515640 1 Processor -- start 2024-03-29T10:46:53.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.134+0000 7f27df515640 1 -- start start 2024-03-29T10:46:53.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.134+0000 7f27df515640 10 monclient: init 2024-03-29T10:46:53.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.135+0000 7f27df515640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:53.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.135+0000 7f27df515640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:53.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.135+0000 7f27df515640 10 monclient(hunting): picked mon.noname-c con 0x7f27d00389e0 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:53.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.135+0000 7f27df515640 10 monclient(hunting): picked mon.noname-a con 0x7f27d0038dc0 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:53.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.135+0000 7f27df515640 10 monclient(hunting): picked mon.noname-b con 0x7f27d0030660 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:53.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.135+0000 7f27df515640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d00c54c0 con 0x7f27d0038dc0 2024-03-29T10:46:53.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.135+0000 7f27df515640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d0036430 con 0x7f27d00389e0 2024-03-29T10:46:53.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.135+0000 7f27df515640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d0039840 con 0x7f27d0030660 2024-03-29T10:46:53.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.135+0000 7f27df515640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:53.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.135+0000 7f27df515640 10 monclient(hunting): authenticate will time out at 1278.376356s 2024-03-29T10:46:53.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.139+0000 7f27dcab0640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f27d0038dc0 0x7f27d00cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:53645/0 (socket says 172.21.15.114:53645) 2024-03-29T10:46:53.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.139+0000 7f27dcab0640 1 -- 172.21.15.114:0/1722426665 learned_addr learned my addr 172.21.15.114:0/1722426665 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:53.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.139+0000 7f27dd2b1640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f27d00389e0 0x7f27d0033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:53148/0 (socket says 172.21.15.114:53148) 2024-03-29T10:46:53.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 1 -- 172.21.15.114:0/1722426665 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729363874 0 0) 0x7f27d0036430 con 0x7f27d00389e0 2024-03-29T10:46:53.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:53.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:53.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:53.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:46:53.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 1 -- 172.21.15.114:0/1722426665 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27c4003520 con 0x7f27d00389e0 2024-03-29T10:46:53.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 1 -- 172.21.15.114:0/1722426665 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709477862 0 0) 0x7f27d0039840 con 0x7f27d0030660 2024-03-29T10:46:53.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:53.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:53.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:53.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:46:53.124 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 1 -- 172.21.15.114:0/1722426665 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d0036430 con 0x7f27d0030660 2024-03-29T10:46:53.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 1 -- 172.21.15.114:0/1722426665 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3516925771 0 0) 0x7f27c4003520 con 0x7f27d00389e0 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 1 -- 172.21.15.114:0/1722426665 >> v1:172.21.15.114:6790/0 conn(0x7f27d0030660 legacy=0x7f27d00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 1 -- 172.21.15.114:0/1722426665 >> v1:172.21.15.114:6789/0 conn(0x7f27d0038dc0 legacy=0x7f27d00cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 1 monclient: found mon.noname-c 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.141+0000 7f27cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 1 -- 172.21.15.114:0/1722426665 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d0035690 con 0x7f27d00389e0 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 10 monclient: _finish_auth 0 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 1 -- 172.21.15.114:0/1722426665 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f27d40015c0 con 0x7f27d00389e0 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:53.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 10 monclient: dump: 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 1 -- 172.21.15.114:0/1722426665 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27d4001850 con 0x7f27d00389e0 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 1 -- 172.21.15.114:0/1722426665 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f27d4001be0 con 0x7f27d00389e0 2024-03-29T10:46:53.128 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27cffff640 10 monclient: dump: 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27df515640 5 monclient: authenticate success, global_id 4477 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27df515640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:53.129 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.142+0000 7f27df515640 4 set_mon_vals no callback set 2024-03-29T10:46:53.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.143+0000 7f27df515640 10 monclient: shutdown 2024-03-29T10:46:53.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.143+0000 7f27df515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:53.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.143+0000 7f27df515640 1 -- 172.21.15.114:0/1722426665 >> v1:172.21.15.176:6789/0 conn(0x7f27d00389e0 legacy=0x7f27d0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.144+0000 7f27df515640 1 -- 172.21.15.114:0/1722426665 shutdown_connections 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.144+0000 7f27df515640 1 -- 172.21.15.114:0/1722426665 >> 172.21.15.114:0/1722426665 conn(0x7f27d0005c00 msgr2=0x7f27d0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.146+0000 7f27df515640 1 -- 172.21.15.114:0/1722426665 shutdown_connections 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.146+0000 7f27df515640 1 -- 172.21.15.114:0/1722426665 wait complete. 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.147+0000 7f27df515640 10 monclient: build_initial_monmap 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.147+0000 7f27df515640 10 monclient: monmap: 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:53.131 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:46:53.132 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:46:53.132 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:46:53.132 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:53.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.148+0000 7f27df515640 1 librados: starting msgr at 2024-03-29T10:46:53.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.148+0000 7f27df515640 1 librados: starting objecter 2024-03-29T10:46:53.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.148+0000 7f27df515640 1 Processor -- start 2024-03-29T10:46:53.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.148+0000 7f27df515640 1 -- start start 2024-03-29T10:46:53.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.148+0000 7f27df515640 1 librados: setting wanted keys 2024-03-29T10:46:53.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.148+0000 7f27df515640 1 librados: calling monclient init 2024-03-29T10:46:53.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.148+0000 7f27df515640 10 monclient: init 2024-03-29T10:46:53.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.150+0000 7f27df515640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:53.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.150+0000 7f27df515640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:53.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.150+0000 7f27df515640 10 monclient(hunting): picked mon.noname-c con 0x7f27d0030660 addr v1:172.21.15.114:6790/0 2024-03-29T10:46:53.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.150+0000 7f27df515640 10 monclient(hunting): picked mon.noname-a con 0x7f27d00389e0 addr v1:172.21.15.114:6789/0 2024-03-29T10:46:53.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.150+0000 7f27df515640 10 monclient(hunting): picked mon.noname-b con 0x7f27d002d630 addr v1:172.21.15.176:6789/0 2024-03-29T10:46:53.136 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.150+0000 7f27df515640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d002dc40 con 0x7f27d00389e0 2024-03-29T10:46:53.136 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.150+0000 7f27df515640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d002f1c0 con 0x7f27d002d630 2024-03-29T10:46:53.136 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.150+0000 7f27df515640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d002eda0 con 0x7f27d0030660 2024-03-29T10:46:53.136 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.150+0000 7f27df515640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:53.136 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.150+0000 7f27df515640 10 monclient(hunting): authenticate will time out at 1278.391698s 2024-03-29T10:46:53.136 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.153+0000 7f27ddab2640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f27d002d630 0x7f27d015ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:33848/0 (socket says 172.21.15.114:33848) 2024-03-29T10:46:53.136 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.153+0000 7f27ddab2640 1 -- 172.21.15.114:0/3502566506 learned_addr learned my addr 172.21.15.114:0/3502566506 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:46:53.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 client.?.objecter ms_handle_connect 0x7f27d00389e0 2024-03-29T10:46:53.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:53.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 client.?.objecter ms_handle_connect 0x7f27d002d630 2024-03-29T10:46:53.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 client.?.objecter ms_handle_connect 0x7f27d0030660 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778965289 0 0) 0x7f27d002f1c0 con 0x7f27d002d630 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27bc004eb0 con 0x7f27d002d630 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175688284 0 0) 0x7f27d002eda0 con 0x7f27d0030660 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:46:53.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.154+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d002f1c0 con 0x7f27d0030660 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3604847159 0 0) 0x7f27bc004eb0 con 0x7f27d002d630 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27d002eda0 con 0x7f27d002d630 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f27d804fdd0 con 0x7f27d002d630 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 10 monclient(hunting): dump: 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:53.141 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3800345854 0 0) 0x7f27d002f1c0 con 0x7f27d0030660 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27bc004eb0 con 0x7f27d0030660 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f27d4001540 con 0x7f27d0030660 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 10 monclient(hunting): dump: 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:53.142 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3717976115 0 0) 0x7f27d002eda0 con 0x7f27d002d630 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 >> v1:172.21.15.114:6790/0 conn(0x7f27d0030660 legacy=0x7f27d00c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 >> v1:172.21.15.114:6789/0 conn(0x7f27d00389e0 legacy=0x7f27d002f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 1 monclient: found mon.b 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.155+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d015c9c0 con 0x7f27d002d630 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.156+0000 7f27cdffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:53.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.156+0000 7f27cdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:53.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.156+0000 7f27cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:53.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.156+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27d804f400 con 0x7f27d002d630 2024-03-29T10:46:53.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.156+0000 7f27cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:53.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.156+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f27d8075f10 con 0x7f27d002d630 2024-03-29T10:46:53.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.156+0000 7f27cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:53.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.156+0000 7f27cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:53.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.156+0000 7f27cdffb640 10 monclient: dump: 2024-03-29T10:46:53.144 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:46:53.144 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:46:53.144 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:53.144 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:46:53.145 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:53.145 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:46:53.145 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:46:53.145 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:46:53.145 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:46:53.145 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:46:53.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.156+0000 7f27cf7fe640 4 set_mon_vals no callback set 2024-03-29T10:46:53.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.158+0000 7f27df515640 5 monclient: authenticate success, global_id 4480 2024-03-29T10:46:53.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.158+0000 7f27df515640 10 monclient: _renew_subs 2024-03-29T10:46:53.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.158+0000 7f27df515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:53.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.158+0000 7f27df515640 1 -- 172.21.15.114:0/3502566506 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d002fdb0 con 0x7f27d002d630 2024-03-29T10:46:53.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.158+0000 7f27df515640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.158+0000 7f27df515640 10 monclient: _renew_subs 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.158+0000 7f27df515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.158+0000 7f27df515640 1 -- 172.21.15.114:0/3502566506 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d0030100 con 0x7f27d002d630 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.158+0000 7f27df515640 1 librados: init done 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.160+0000 7f27df515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.160+0000 7f27df515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.160+0000 7f27df515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.160+0000 7f27df515640 1 -- 172.21.15.114:0/3502566506 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d015d960 con 0x7f27d002d630 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.161+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f27d8067bf0 con 0x7f27d002d630 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.162+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f27d80ff3b0 con 0x7f27d002d630 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.162+0000 7f27cdffb640 10 client.4480.objecter ms_dispatch 0x7f27d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.162+0000 7f27cdffb640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.162+0000 7f27cdffb640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:53.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.162+0000 7f27cdffb640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-29T10:46:53.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.165+0000 7f27dd2b1640 10 monclient: get_auth_request con 0x7f27bc06dd20 auth_method 0 2024-03-29T10:46:53.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.166+0000 7f27cdffb640 10 client.4480.objecter ms_handle_connect 0x7f27bc06dd20 2024-03-29T10:46:53.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.168+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27d80cac90 con 0x7f27d002d630 2024-03-29T10:46:53.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.168+0000 7f27cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:53.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.168+0000 7f27cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:53.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.503+0000 7f27df515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:53.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.503+0000 7f27df515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:53.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.503+0000 7f27df515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:46:53.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.503+0000 7f27df515640 1 -- 172.21.15.114:0/3502566506 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27d0007b90 con 0x7f27d002d630 2024-03-29T10:46:53.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.506+0000 7f27cdffb640 1 -- 172.21.15.114:0/3502566506 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f27d80ccbc0 con 0x7f27d002d630 2024-03-29T10:46:53.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.506+0000 7f27cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:53.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.506+0000 7f27cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:53.489 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:46:53.489 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:53.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.510+0000 7f27bb7fe640 10 librados: watch_flush enter 2024-03-29T10:46:53.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.510+0000 7f27bb7fe640 10 librados: watch_flush exit 2024-03-29T10:46:53.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.510+0000 7f27bb7fe640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-29T10:46:53.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.510+0000 7f27bb7fe640 10 client.4480.objecter successfully canceled tick 2024-03-29T10:46:53.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.510+0000 7f27bb7fe640 1 -- 172.21.15.114:0/3502566506 >> v1:172.21.15.114:6816/775044307 conn(0x7f27bc06dd20 legacy=0x7f27bc0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:53.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.510+0000 7f27bb7fe640 10 monclient: shutdown 2024-03-29T10:46:53.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.510+0000 7f27bb7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:53.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.510+0000 7f27bb7fe640 1 -- 172.21.15.114:0/3502566506 >> v1:172.21.15.176:6789/0 conn(0x7f27d002d630 legacy=0x7f27d015ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:53.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.511+0000 7f27bb7fe640 1 -- 172.21.15.114:0/3502566506 shutdown_connections 2024-03-29T10:46:53.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.511+0000 7f27bb7fe640 1 -- 172.21.15.114:0/3502566506 >> 172.21.15.114:0/3502566506 conn(0x7f27d0005c00 msgr2=0x7f27d0034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:53.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.512+0000 7f27bb7fe640 1 -- 172.21.15.114:0/3502566506 shutdown_connections 2024-03-29T10:46:53.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.512+0000 7f27bb7fe640 1 -- 172.21.15.114:0/3502566506 wait complete. 2024-03-29T10:46:53.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:46:53.513+0000 7f27bb7fe640 1 librados: shutdown 2024-03-29T10:46:53.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:00.320 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:00.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.481+0000 7fcb5f76b640 10 monclient: get_monmap_and_config 2024-03-29T10:47:00.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.481+0000 7fcb5f76b640 10 monclient: build_initial_monmap 2024-03-29T10:47:00.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.481+0000 7fcb5f76b640 10 monclient: monmap: 2024-03-29T10:47:00.464 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:00.464 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:00.464 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:47:00.482526+0000 2024-03-29T10:47:00.464 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:47:00.482526+0000 2024-03-29T10:47:00.464 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:00.464 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:00.465 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:00.465 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:47:00.465 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:47:00.465 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:00.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.482+0000 7fcb5f76b640 1 Processor -- start 2024-03-29T10:47:00.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.482+0000 7fcb5f76b640 1 -- start start 2024-03-29T10:47:00.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.482+0000 7fcb5f76b640 10 monclient: init 2024-03-29T10:47:00.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5f76b640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:00.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5f76b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:00.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5f76b640 10 monclient(hunting): picked mon.noname-b con 0x7fcb581422b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:00.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5f76b640 10 monclient(hunting): picked mon.noname-a con 0x7fcb58142690 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:00.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5f76b640 10 monclient(hunting): picked mon.noname-c con 0x7fcb580b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:00.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5f76b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb58142c80 con 0x7fcb58142690 2024-03-29T10:47:00.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5f76b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb580aae40 con 0x7fcb580b17d0 2024-03-29T10:47:00.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5f76b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5813fbd0 con 0x7fcb581422b0 2024-03-29T10:47:00.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5f76b640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:00.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5f76b640 10 monclient(hunting): authenticate will time out at 1285.724381s 2024-03-29T10:47:00.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5dd08640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fcb580b17d0 0x7fcb580b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:47258/0 (socket says 172.21.15.114:47258) 2024-03-29T10:47:00.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb5dd08640 1 -- 172.21.15.114:0/2529137668 learned_addr learned my addr 172.21.15.114:0/2529137668 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:00.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb4ffff640 1 -- 172.21.15.114:0/2529137668 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912722184 0 0) 0x7fcb580aae40 con 0x7fcb580b17d0 2024-03-29T10:47:00.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb4ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:00.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:00.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:00.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb4ffff640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:47:00.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.483+0000 7fcb4ffff640 1 -- 172.21.15.114:0/2529137668 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb44003520 con 0x7fcb580b17d0 2024-03-29T10:47:00.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 1 -- 172.21.15.114:0/2529137668 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298462017 0 0) 0x7fcb5813fbd0 con 0x7fcb581422b0 2024-03-29T10:47:00.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:00.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:00.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:00.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 10 monclient(hunting): my global_id is 4571 2024-03-29T10:47:00.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 1 -- 172.21.15.114:0/2529137668 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb580aae40 con 0x7fcb581422b0 2024-03-29T10:47:00.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 1 -- 172.21.15.114:0/2529137668 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157296086 0 0) 0x7fcb44003520 con 0x7fcb580b17d0 2024-03-29T10:47:00.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 1 -- 172.21.15.114:0/2529137668 >> v1:172.21.15.114:6790/0 conn(0x7fcb581422b0 legacy=0x7fcb5813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 1 -- 172.21.15.114:0/2529137668 >> v1:172.21.15.114:6789/0 conn(0x7fcb58142690 legacy=0x7fcb580b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:00.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 1 monclient: found mon.noname-c 2024-03-29T10:47:00.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:00.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:47:00.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 1 -- 172.21.15.114:0/2529137668 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb58151940 con 0x7fcb580b17d0 2024-03-29T10:47:00.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 10 monclient: _finish_auth 0 2024-03-29T10:47:00.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:00.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:00.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb5f76b640 5 monclient: authenticate success, global_id 4483 2024-03-29T10:47:00.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb5f76b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:00.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 1 -- 172.21.15.114:0/2529137668 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fcb50001a10 con 0x7fcb580b17d0 2024-03-29T10:47:00.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb4ffff640 10 monclient: dump: 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:00.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.484+0000 7fcb5f76b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:00.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.485+0000 7fcb4ffff640 1 -- 172.21.15.114:0/2529137668 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb50001b70 con 0x7fcb580b17d0 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.485+0000 7fcb4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.485+0000 7fcb5f76b640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.485+0000 7fcb4ffff640 1 -- 172.21.15.114:0/2529137668 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fcb50001f40 con 0x7fcb580b17d0 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.486+0000 7fcb5f76b640 4 set_mon_vals no callback set 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.486+0000 7fcb4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.486+0000 7fcb4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.486+0000 7fcb4ffff640 10 monclient: dump: 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:00.483 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.486+0000 7fcb5f76b640 10 monclient: shutdown 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.486+0000 7fcb5f76b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.486+0000 7fcb5f76b640 1 -- 172.21.15.114:0/2529137668 >> v1:172.21.15.176:6789/0 conn(0x7fcb580b17d0 legacy=0x7fcb580b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.487+0000 7fcb5f76b640 1 -- 172.21.15.114:0/2529137668 shutdown_connections 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.487+0000 7fcb5f76b640 1 -- 172.21.15.114:0/2529137668 >> 172.21.15.114:0/2529137668 conn(0x7fcb58066a00 msgr2=0x7fcb58066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.487+0000 7fcb5f76b640 1 -- 172.21.15.114:0/2529137668 shutdown_connections 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.487+0000 7fcb5f76b640 1 -- 172.21.15.114:0/2529137668 wait complete. 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.488+0000 7fcb5f76b640 10 monclient: build_initial_monmap 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.488+0000 7fcb5f76b640 10 monclient: monmap: 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:00.484 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.488+0000 7fcb5f76b640 1 librados: starting msgr at 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.488+0000 7fcb5f76b640 1 librados: starting objecter 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.488+0000 7fcb5f76b640 1 Processor -- start 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.488+0000 7fcb5f76b640 1 -- start start 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.488+0000 7fcb5f76b640 1 librados: setting wanted keys 2024-03-29T10:47:00.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.488+0000 7fcb5f76b640 1 librados: calling monclient init 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.488+0000 7fcb5f76b640 10 monclient: init 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.489+0000 7fcb5f76b640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.489+0000 7fcb5f76b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.489+0000 7fcb5f76b640 10 monclient(hunting): picked mon.noname-a con 0x7fcb580b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.489+0000 7fcb5f76b640 10 monclient(hunting): picked mon.noname-c con 0x7fcb581422b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.489+0000 7fcb5f76b640 10 monclient(hunting): picked mon.noname-b con 0x7fcb58142690 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.489+0000 7fcb5f76b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb580ab730 con 0x7fcb580b17d0 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.489+0000 7fcb5f76b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb581dfe50 con 0x7fcb58142690 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb5f76b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb580ab980 con 0x7fcb581422b0 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb5f76b640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb5f76b640 10 monclient(hunting): authenticate will time out at 1285.731018s 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb5cd06640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fcb581422b0 0x7fcb5812b630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55338/0 (socket says 172.21.15.114:55338) 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb5cd06640 1 -- 172.21.15.114:0/4184762206 learned_addr learned my addr 172.21.15.114:0/4184762206 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:00.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 client.?.objecter ms_handle_connect 0x7fcb581422b0 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 client.?.objecter ms_handle_connect 0x7fcb58142690 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 client.?.objecter ms_handle_connect 0x7fcb580b17d0 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729459112 0 0) 0x7fcb580ab730 con 0x7fcb580b17d0 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 monclient(hunting): my global_id is 4512 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb3c004eb0 con 0x7fcb580b17d0 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239975570 0 0) 0x7fcb580ab980 con 0x7fcb581422b0 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:00.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 10 monclient(hunting): my global_id is 4574 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.490+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb580ab730 con 0x7fcb581422b0 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3558591228 0 0) 0x7fcb580ab730 con 0x7fcb581422b0 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb580ab980 con 0x7fcb581422b0 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fcb480018f0 con 0x7fcb581422b0 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 10 monclient(hunting): dump: 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:00.488 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254826356 0 0) 0x7fcb581dfe50 con 0x7fcb58142690 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb580ab730 con 0x7fcb58142690 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2416976155 0 0) 0x7fcb3c004eb0 con 0x7fcb580b17d0 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.491+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb581dfe50 con 0x7fcb580b17d0 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fcb54001930 con 0x7fcb580b17d0 2024-03-29T10:47:00.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 10 monclient(hunting): dump: 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2219029573 0 0) 0x7fcb580ab980 con 0x7fcb581422b0 2024-03-29T10:47:00.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 >> v1:172.21.15.176:6789/0 conn(0x7fcb58142690 legacy=0x7fcb581e6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 >> v1:172.21.15.114:6789/0 conn(0x7fcb580b17d0 legacy=0x7fcb581df260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 1 monclient: found mon.c 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb580abc60 con 0x7fcb581422b0 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb5f76b640 5 monclient: authenticate success, global_id 4574 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb5f76b640 10 monclient: _renew_subs 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb5f76b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb5f76b640 1 -- 172.21.15.114:0/4184762206 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb581df970 con 0x7fcb581422b0 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb5f76b640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:00.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb5f76b640 10 monclient: _renew_subs 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb5f76b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb5f76b640 1 -- 172.21.15.114:0/4184762206 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb581dfc20 con 0x7fcb581422b0 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb5f76b640 1 librados: init done 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb48001a50 con 0x7fcb581422b0 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fcb48003830 con 0x7fcb581422b0 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.492+0000 7fcb4dffb640 10 monclient: dump: 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:00.492 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.493+0000 7fcb4f7fe640 4 set_mon_vals no callback set 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.493+0000 7fcb5f76b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.494+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fcb48003a70 con 0x7fcb581422b0 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.494+0000 7fcb5f76b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.495+0000 7fcb5f76b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.495+0000 7fcb5f76b640 1 -- 172.21.15.114:0/4184762206 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb38005080 con 0x7fcb581422b0 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.495+0000 7fcb5d507640 10 monclient: get_auth_request con 0x7fcb3c06dfb0 auth_method 0 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.495+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fcb480849a0 con 0x7fcb581422b0 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.495+0000 7fcb4dffb640 10 client.4574.objecter ms_dispatch 0x7fcb58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:00.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.495+0000 7fcb4dffb640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:00.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.495+0000 7fcb4dffb640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:00.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.495+0000 7fcb4dffb640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T10:47:00.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.496+0000 7fcb4dffb640 10 client.4574.objecter ms_handle_connect 0x7fcb3c06dfb0 2024-03-29T10:47:00.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.503+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb48050280 con 0x7fcb581422b0 2024-03-29T10:47:00.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.503+0000 7fcb4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:00.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.503+0000 7fcb4dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:00.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.858+0000 7fcb5f76b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:00.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.858+0000 7fcb5f76b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:00.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.858+0000 7fcb5f76b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:00.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.858+0000 7fcb5f76b640 1 -- 172.21.15.114:0/4184762206 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb38005330 con 0x7fcb581422b0 2024-03-29T10:47:00.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.859+0000 7fcb4dffb640 1 -- 172.21.15.114:0/4184762206 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fcb480521b0 con 0x7fcb581422b0 2024-03-29T10:47:00.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.859+0000 7fcb4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:00.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.859+0000 7fcb4dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:00.842 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:47:00.843 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:00.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.863+0000 7fcb5f76b640 10 librados: watch_flush enter 2024-03-29T10:47:00.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.863+0000 7fcb5f76b640 10 librados: watch_flush exit 2024-03-29T10:47:00.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.863+0000 7fcb5f76b640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T10:47:00.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.863+0000 7fcb5f76b640 10 client.4574.objecter successfully canceled tick 2024-03-29T10:47:00.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.863+0000 7fcb5f76b640 1 -- 172.21.15.114:0/4184762206 >> v1:172.21.15.114:6816/775044307 conn(0x7fcb3c06dfb0 legacy=0x7fcb3c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.863+0000 7fcb5f76b640 10 monclient: shutdown 2024-03-29T10:47:00.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.863+0000 7fcb5f76b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:00.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.863+0000 7fcb5f76b640 1 -- 172.21.15.114:0/4184762206 >> v1:172.21.15.114:6790/0 conn(0x7fcb581422b0 legacy=0x7fcb5812b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.863+0000 7fcb5f76b640 1 -- 172.21.15.114:0/4184762206 shutdown_connections 2024-03-29T10:47:00.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.863+0000 7fcb5f76b640 1 -- 172.21.15.114:0/4184762206 >> 172.21.15.114:0/4184762206 conn(0x7fcb58066a00 msgr2=0x7fcb5813c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:00.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.863+0000 7fcb5f76b640 1 -- 172.21.15.114:0/4184762206 shutdown_connections 2024-03-29T10:47:00.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.864+0000 7fcb5f76b640 1 -- 172.21.15.114:0/4184762206 wait complete. 2024-03-29T10:47:00.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:00.864+0000 7fcb5f76b640 1 librados: shutdown 2024-03-29T10:47:00.861 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:07.668 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:07.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.835+0000 7f7b0ffff640 10 monclient: get_monmap_and_config 2024-03-29T10:47:07.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.835+0000 7f7b0ffff640 10 monclient: build_initial_monmap 2024-03-29T10:47:07.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.835+0000 7f7b0ffff640 10 monclient: monmap: 2024-03-29T10:47:07.818 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:07.818 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:07.818 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:47:07.836651+0000 2024-03-29T10:47:07.818 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:47:07.836651+0000 2024-03-29T10:47:07.819 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:07.819 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:07.819 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:07.819 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:47:07.819 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:47:07.819 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:07.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.836+0000 7f7b0ffff640 1 Processor -- start 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.836+0000 7f7b0ffff640 1 -- start start 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.836+0000 7f7b0ffff640 10 monclient: init 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.837+0000 7f7b0ffff640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.837+0000 7f7b0ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.837+0000 7f7b0ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f7b080389e0 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.837+0000 7f7b0ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f7b08038dc0 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.837+0000 7f7b0ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f7b08030660 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.837+0000 7f7b0ffff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080c54c0 con 0x7f7b080389e0 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.837+0000 7f7b0ffff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b08036430 con 0x7f7b08030660 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.837+0000 7f7b0ffff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b08039840 con 0x7f7b08038dc0 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.837+0000 7f7b0ffff640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:07.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.837+0000 7f7b0ffff640 10 monclient(hunting): authenticate will time out at 1293.078283s 2024-03-29T10:47:07.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.840+0000 7f7b0e7fc640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f7b08038dc0 0x7f7b080cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:45070/0 (socket says 172.21.15.114:45070) 2024-03-29T10:47:07.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.840+0000 7f7b0e7fc640 1 -- 172.21.15.114:0/841206518 learned_addr learned my addr 172.21.15.114:0/841206518 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:07.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 1 -- 172.21.15.114:0/841206518 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892356194 0 0) 0x7f7b08039840 con 0x7f7b08038dc0 2024-03-29T10:47:07.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:07.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 10 monclient(hunting): my global_id is 4577 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 1 -- 172.21.15.114:0/841206518 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af8003520 con 0x7f7b08038dc0 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 1 -- 172.21.15.114:0/841206518 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3781856724 0 0) 0x7f7af8003520 con 0x7f7b08038dc0 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 1 -- 172.21.15.114:0/841206518 >> v1:172.21.15.176:6789/0 conn(0x7f7b08030660 legacy=0x7f7b080d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 1 -- 172.21.15.114:0/841206518 >> v1:172.21.15.114:6789/0 conn(0x7f7b080389e0 legacy=0x7f7b08033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 1 monclient: found mon.noname-b 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 1 -- 172.21.15.114:0/841206518 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b08035690 con 0x7f7b08038dc0 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:07.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 1 -- 172.21.15.114:0/841206518 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7b00001690 con 0x7f7b08038dc0 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 10 monclient: dump: 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:07.826 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0ffff640 5 monclient: authenticate success, global_id 4577 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.841+0000 7f7b0dffb640 1 -- 172.21.15.114:0/841206518 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b00001860 con 0x7f7b08038dc0 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.842+0000 7f7b0dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.842+0000 7f7b0dffb640 1 -- 172.21.15.114:0/841206518 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7b00001cb0 con 0x7f7b08038dc0 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.842+0000 7f7b0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.842+0000 7f7b0dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.842+0000 7f7b0dffb640 10 monclient: dump: 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:07.827 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:07.828 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:07.828 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:07.828 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:07.828 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:07.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.842+0000 7f7b0ffff640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:07.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.842+0000 7f7b0ffff640 4 set_mon_vals no callback set 2024-03-29T10:47:07.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.842+0000 7f7b0ffff640 10 monclient: shutdown 2024-03-29T10:47:07.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.842+0000 7f7b0ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.842+0000 7f7b0ffff640 1 -- 172.21.15.114:0/841206518 >> v1:172.21.15.114:6790/0 conn(0x7f7b08038dc0 legacy=0x7f7b080cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.843+0000 7f7b0ffff640 1 -- 172.21.15.114:0/841206518 shutdown_connections 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.843+0000 7f7b0ffff640 1 -- 172.21.15.114:0/841206518 >> 172.21.15.114:0/841206518 conn(0x7f7b08005c00 msgr2=0x7f7b08006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.843+0000 7f7b0ffff640 1 -- 172.21.15.114:0/841206518 shutdown_connections 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.843+0000 7f7b0ffff640 1 -- 172.21.15.114:0/841206518 wait complete. 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.844+0000 7f7b0ffff640 10 monclient: build_initial_monmap 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.844+0000 7f7b0ffff640 10 monclient: monmap: 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:07.830 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:07.831 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:47:07.831 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:47:07.831 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:07.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.844+0000 7f7b0ffff640 1 librados: starting msgr at 2024-03-29T10:47:07.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.844+0000 7f7b0ffff640 1 librados: starting objecter 2024-03-29T10:47:07.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.845+0000 7f7b0ffff640 1 Processor -- start 2024-03-29T10:47:07.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.845+0000 7f7b0ffff640 1 -- start start 2024-03-29T10:47:07.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.845+0000 7f7b0ffff640 1 librados: setting wanted keys 2024-03-29T10:47:07.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.845+0000 7f7b0ffff640 1 librados: calling monclient init 2024-03-29T10:47:07.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.845+0000 7f7b0ffff640 10 monclient: init 2024-03-29T10:47:07.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.845+0000 7f7b0ffff640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:07.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.845+0000 7f7b0ffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:07.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.845+0000 7f7b0ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f7b08030660 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:07.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.847+0000 7f7b0effd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f7b08030660 0x7f7b080c5da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:47178/0 (socket says 172.21.15.114:47178) 2024-03-29T10:47:07.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.848+0000 7f7b0ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f7b080389e0 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:07.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.848+0000 7f7b0effd640 1 -- 172.21.15.114:0/1917842636 learned_addr learned my addr 172.21.15.114:0/1917842636 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:07.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 10 client.?.objecter ms_handle_connect 0x7f7b08030660 2024-03-29T10:47:07.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:07.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7b0ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f7b080c64b0 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:07.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7b0ffff640 1 -- 172.21.15.114:0/1917842636 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080c2400 con 0x7f7b080389e0 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7b0ffff640 1 -- 172.21.15.114:0/1917842636 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0800d720 con 0x7f7b080c64b0 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7b0ffff640 1 -- 172.21.15.114:0/1917842636 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0800a1f0 con 0x7f7b08030660 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7b0ffff640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7b0ffff640 10 monclient(hunting): authenticate will time out at 1293.090301s 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982474579 0 0) 0x7f7b0800a1f0 con 0x7f7b08030660 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 10 monclient(hunting): my global_id is 4580 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7afc004eb0 con 0x7f7b08030660 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2781742914 0 0) 0x7f7afc004eb0 con 0x7f7b08030660 2024-03-29T10:47:07.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b0800a1f0 con 0x7f7b08030660 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7b04001930 con 0x7f7b08030660 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.849+0000 7f7af77fe640 10 monclient(hunting): dump: 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:07.835 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2843698636 0 0) 0x7f7b0800a1f0 con 0x7f7b08030660 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 >> v1:172.21.15.176:6789/0 conn(0x7f7b080c64b0 legacy=0x7f7b080c1cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 >> v1:172.21.15.114:6789/0 conn(0x7f7b080389e0 legacy=0x7f7b080c15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 1 monclient: found mon.c 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0800a4d0 con 0x7f7b08030660 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7af77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7b0ffff640 5 monclient: authenticate success, global_id 4580 2024-03-29T10:47:07.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7b0ffff640 10 monclient: _renew_subs 2024-03-29T10:47:07.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7b0ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:07.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7b0ffff640 1 -- 172.21.15.114:0/1917842636 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b080c5880 con 0x7f7b08030660 2024-03-29T10:47:07.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7b0ffff640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:07.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7b0ffff640 10 monclient: _renew_subs 2024-03-29T10:47:07.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7b0ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:07.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7b0ffff640 1 -- 172.21.15.114:0/1917842636 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b080c2680 con 0x7f7b08030660 2024-03-29T10:47:07.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.850+0000 7f7b0ffff640 1 librados: init done 2024-03-29T10:47:07.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.852+0000 7f7af57fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:07.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.852+0000 7f7af57fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:07.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.852+0000 7f7af57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:07.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.852+0000 7f7af57fa640 1 -- 172.21.15.114:0/1917842636 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b08007b90 con 0x7f7b08030660 2024-03-29T10:47:07.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.853+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b04003610 con 0x7f7b08030660 2024-03-29T10:47:07.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.853+0000 7f7af77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:07.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.853+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7b04003a60 con 0x7f7b08030660 2024-03-29T10:47:07.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.853+0000 7f7af77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:07.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.853+0000 7f7af77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:07.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.853+0000 7f7af77fe640 10 monclient: dump: 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.853+0000 7f7b0cff9640 4 set_mon_vals no callback set 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.854+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f7b04003c80 con 0x7f7b08030660 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.855+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f7b04084a10 con 0x7f7b08030660 2024-03-29T10:47:07.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.855+0000 7f7b0e7fc640 10 monclient: get_auth_request con 0x7f7afc06dce0 auth_method 0 2024-03-29T10:47:07.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.856+0000 7f7af77fe640 10 client.4580.objecter ms_dispatch 0x7f7b10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:07.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.856+0000 7f7af77fe640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:07.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.856+0000 7f7af77fe640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:07.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.856+0000 7f7af77fe640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T10:47:07.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.856+0000 7f7af77fe640 10 client.4580.objecter ms_handle_connect 0x7f7afc06dce0 2024-03-29T10:47:07.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.862+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b040502f0 con 0x7f7b08030660 2024-03-29T10:47:07.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.862+0000 7f7af77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:07.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:07.862+0000 7f7af77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:08.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.190+0000 7f7af57fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:08.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.190+0000 7f7af57fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:08.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.190+0000 7f7af57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:08.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.190+0000 7f7af57fa640 1 -- 172.21.15.114:0/1917842636 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b080c56e0 con 0x7f7b08030660 2024-03-29T10:47:08.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.191+0000 7f7af77fe640 1 -- 172.21.15.114:0/1917842636 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f7b080c56e0 con 0x7f7b08030660 2024-03-29T10:47:08.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.191+0000 7f7af77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:08.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.191+0000 7f7af77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:08.175 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:47:08.175 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:08.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.194+0000 7f7af57fa640 10 librados: watch_flush enter 2024-03-29T10:47:08.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.195+0000 7f7af57fa640 10 librados: watch_flush exit 2024-03-29T10:47:08.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.195+0000 7f7af57fa640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T10:47:08.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.195+0000 7f7af57fa640 10 client.4580.objecter successfully canceled tick 2024-03-29T10:47:08.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.195+0000 7f7af57fa640 1 -- 172.21.15.114:0/1917842636 >> v1:172.21.15.114:6816/775044307 conn(0x7f7afc06dce0 legacy=0x7f7afc0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:08.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.195+0000 7f7af57fa640 10 monclient: shutdown 2024-03-29T10:47:08.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.195+0000 7f7af57fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:08.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.195+0000 7f7af57fa640 1 -- 172.21.15.114:0/1917842636 >> v1:172.21.15.114:6790/0 conn(0x7f7b08030660 legacy=0x7f7b080c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:08.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.196+0000 7f7af57fa640 1 -- 172.21.15.114:0/1917842636 shutdown_connections 2024-03-29T10:47:08.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.196+0000 7f7af57fa640 1 -- 172.21.15.114:0/1917842636 >> 172.21.15.114:0/1917842636 conn(0x7f7b08005c00 msgr2=0x7f7b08034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:08.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.196+0000 7f7af57fa640 1 -- 172.21.15.114:0/1917842636 shutdown_connections 2024-03-29T10:47:08.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.196+0000 7f7af57fa640 1 -- 172.21.15.114:0/1917842636 wait complete. 2024-03-29T10:47:08.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:08.196+0000 7f7af57fa640 1 librados: shutdown 2024-03-29T10:47:08.196 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:15.003 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:15.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.174+0000 7f6b482d4640 10 monclient: get_monmap_and_config 2024-03-29T10:47:15.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.174+0000 7f6b482d4640 10 monclient: build_initial_monmap 2024-03-29T10:47:15.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.174+0000 7f6b482d4640 10 monclient: monmap: 2024-03-29T10:47:15.157 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:15.157 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:15.157 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:47:15.175374+0000 2024-03-29T10:47:15.157 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:47:15.175374+0000 2024-03-29T10:47:15.157 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:15.157 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:15.157 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:15.158 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:47:15.158 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:47:15.158 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.176+0000 7f6b482d4640 1 Processor -- start 2024-03-29T10:47:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.177+0000 7f6b482d4640 1 -- start start 2024-03-29T10:47:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.177+0000 7f6b482d4640 10 monclient: init 2024-03-29T10:47:15.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.178+0000 7f6b482d4640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:15.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.178+0000 7f6b482d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:15.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.178+0000 7f6b482d4640 10 monclient(hunting): picked mon.noname-b con 0x7f6b400b0770 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:15.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.178+0000 7f6b482d4640 10 monclient(hunting): picked mon.noname-a con 0x7f6b4014a700 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:15.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.178+0000 7f6b46070640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f6b400b0770 0x7f6b40149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:59348/0 (socket says 172.21.15.114:59348) 2024-03-29T10:47:15.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.181+0000 7f6b482d4640 10 monclient(hunting): picked mon.noname-c con 0x7f6b400ac5a0 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:15.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.181+0000 7f6b482d4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b400aba70 con 0x7f6b4014a700 2024-03-29T10:47:15.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.181+0000 7f6b482d4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b400aecb0 con 0x7f6b400ac5a0 2024-03-29T10:47:15.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.181+0000 7f6b482d4640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b400af150 con 0x7f6b400b0770 2024-03-29T10:47:15.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.181+0000 7f6b482d4640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:15.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.181+0000 7f6b482d4640 10 monclient(hunting): authenticate will time out at 1300.422676s 2024-03-29T10:47:15.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.181+0000 7f6b46070640 1 -- 172.21.15.114:0/4013745589 learned_addr learned my addr 172.21.15.114:0/4013745589 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:15.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.182+0000 7f6b4506e640 1 -- 172.21.15.114:0/4013745589 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716011698 0 0) 0x7f6b400af150 con 0x7f6b400b0770 2024-03-29T10:47:15.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.182+0000 7f6b4506e640 10 monclient(hunting): _negotiate 2024-03-29T10:47:15.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.182+0000 7f6b4506e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.182+0000 7f6b4506e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.182+0000 7f6b4506e640 10 monclient(hunting): my global_id is 4583 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.182+0000 7f6b4506e640 1 -- 172.21.15.114:0/4013745589 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b2c003520 con 0x7f6b400b0770 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 1 -- 172.21.15.114:0/4013745589 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1823026554 0 0) 0x7f6b2c003520 con 0x7f6b400b0770 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 1 -- 172.21.15.114:0/4013745589 >> v1:172.21.15.176:6789/0 conn(0x7f6b400ac5a0 legacy=0x7f6b40151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 1 -- 172.21.15.114:0/4013745589 >> v1:172.21.15.114:6789/0 conn(0x7f6b4014a700 legacy=0x7f6b400abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 10 monclient: _finish_hunting 0 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 1 monclient: found mon.noname-b 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 1 -- 172.21.15.114:0/4013745589 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b400ac9d0 con 0x7f6b400b0770 2024-03-29T10:47:15.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 10 monclient: _finish_auth 0 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 10 monclient: _check_auth_tickets 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 1 -- 172.21.15.114:0/4013745589 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6b30001690 con 0x7f6b400b0770 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.183+0000 7f6b4506e640 10 monclient: dump: 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:15.170 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:15.171 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:15.171 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:15.171 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:15.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.184+0000 7f6b482d4640 5 monclient: authenticate success, global_id 4583 2024-03-29T10:47:15.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.184+0000 7f6b482d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:15.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.184+0000 7f6b4506e640 1 -- 172.21.15.114:0/4013745589 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b300018f0 con 0x7f6b400b0770 2024-03-29T10:47:15.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.184+0000 7f6b4506e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:15.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.184+0000 7f6b482d4640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:15.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.184+0000 7f6b482d4640 4 set_mon_vals no callback set 2024-03-29T10:47:15.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.185+0000 7f6b4506e640 1 -- 172.21.15.114:0/4013745589 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6b30001d00 con 0x7f6b400b0770 2024-03-29T10:47:15.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.185+0000 7f6b4506e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.185+0000 7f6b4506e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.185+0000 7f6b4506e640 10 monclient: dump: 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.185+0000 7f6b482d4640 10 monclient: shutdown 2024-03-29T10:47:15.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.185+0000 7f6b482d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:15.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.185+0000 7f6b482d4640 1 -- 172.21.15.114:0/4013745589 >> v1:172.21.15.114:6790/0 conn(0x7f6b400b0770 legacy=0x7f6b40149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:15.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.189+0000 7f6b482d4640 1 -- 172.21.15.114:0/4013745589 shutdown_connections 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.189+0000 7f6b482d4640 1 -- 172.21.15.114:0/4013745589 >> 172.21.15.114:0/4013745589 conn(0x7f6b4006b660 msgr2=0x7f6b4006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.189+0000 7f6b482d4640 1 -- 172.21.15.114:0/4013745589 shutdown_connections 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.189+0000 7f6b482d4640 1 -- 172.21.15.114:0/4013745589 wait complete. 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.190+0000 7f6b482d4640 10 monclient: build_initial_monmap 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.190+0000 7f6b482d4640 10 monclient: monmap: 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:47:15.174 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:47:15.175 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:15.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.191+0000 7f6b482d4640 1 librados: starting msgr at 2024-03-29T10:47:15.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.191+0000 7f6b482d4640 1 librados: starting objecter 2024-03-29T10:47:15.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.191+0000 7f6b482d4640 1 Processor -- start 2024-03-29T10:47:15.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.192+0000 7f6b482d4640 1 -- start start 2024-03-29T10:47:15.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.192+0000 7f6b482d4640 1 librados: setting wanted keys 2024-03-29T10:47:15.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.192+0000 7f6b482d4640 1 librados: calling monclient init 2024-03-29T10:47:15.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.192+0000 7f6b482d4640 10 monclient: init 2024-03-29T10:47:15.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.193+0000 7f6b482d4640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:15.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.193+0000 7f6b482d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:15.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.193+0000 7f6b482d4640 10 monclient(hunting): picked mon.noname-c con 0x7f6b400ac5a0 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:15.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.193+0000 7f6b482d4640 10 monclient(hunting): picked mon.noname-a con 0x7f6b400b0770 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:15.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.193+0000 7f6b482d4640 10 monclient(hunting): picked mon.noname-b con 0x7f6b4014a700 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:15.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.194+0000 7f6b482d4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b40126380 con 0x7f6b400b0770 2024-03-29T10:47:15.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.194+0000 7f6b482d4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b40145bc0 con 0x7f6b4014a700 2024-03-29T10:47:15.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.194+0000 7f6b482d4640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b401239d0 con 0x7f6b400ac5a0 2024-03-29T10:47:15.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.194+0000 7f6b482d4640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:15.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.194+0000 7f6b482d4640 10 monclient(hunting): authenticate will time out at 1300.435965s 2024-03-29T10:47:15.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.197+0000 7f6b46070640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f6b400ac5a0 0x7f6b40126810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:36794/0 (socket says 172.21.15.114:36794) 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.197+0000 7f6b46070640 1 -- 172.21.15.114:0/1923185979 learned_addr learned my addr 172.21.15.114:0/1923185979 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.197+0000 7f6b36ffd640 10 client.?.objecter ms_handle_connect 0x7f6b400b0770 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 10 client.?.objecter ms_handle_connect 0x7f6b400ac5a0 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730365367 0 0) 0x7f6b40126380 con 0x7f6b400b0770 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 10 monclient(hunting): my global_id is 4515 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b24004eb0 con 0x7f6b400b0770 2024-03-29T10:47:15.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197205532 0 0) 0x7f6b401239d0 con 0x7f6b400ac5a0 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 10 monclient(hunting): my global_id is 4586 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.198+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b40126380 con 0x7f6b400ac5a0 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.199+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3014737298 0 0) 0x7f6b40126380 con 0x7f6b400ac5a0 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.199+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b401239d0 con 0x7f6b400ac5a0 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.199+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6b30001650 con 0x7f6b400ac5a0 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.199+0000 7f6b36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.199+0000 7f6b36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.199+0000 7f6b36ffd640 10 monclient(hunting): dump: 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:15.186 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.199+0000 7f6b36ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.199+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2246065345 0 0) 0x7f6b24004eb0 con 0x7f6b400b0770 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.199+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b40126380 con 0x7f6b400b0770 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6b38001b50 con 0x7f6b400b0770 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:15.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 10 monclient(hunting): dump: 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4276120171 0 0) 0x7f6b40126380 con 0x7f6b400b0770 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 >> v1:172.21.15.114:6790/0 conn(0x7f6b400ac5a0 legacy=0x7f6b40126810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 >> v1:172.21.15.176:6789/0 conn(0x7f6b4014a700 legacy=0x7f6b401e6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:15.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 1 monclient: found mon.a 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b40123b60 con 0x7f6b400b0770 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 10 monclient: _finish_auth 0 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b482d4640 5 monclient: authenticate success, global_id 4515 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b482d4640 10 monclient: _renew_subs 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b482d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b482d4640 1 -- 172.21.15.114:0/1923185979 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b40126f20 con 0x7f6b400b0770 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b482d4640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b482d4640 10 monclient: _renew_subs 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b482d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:15.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b482d4640 1 -- 172.21.15.114:0/1923185979 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b401dc280 con 0x7f6b400b0770 2024-03-29T10:47:15.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.200+0000 7f6b482d4640 1 librados: init done 2024-03-29T10:47:15.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.203+0000 7f6b482d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:15.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.203+0000 7f6b482d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:15.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.203+0000 7f6b482d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:15.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.203+0000 7f6b482d4640 1 -- 172.21.15.114:0/1923185979 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b400ab560 con 0x7f6b400b0770 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.203+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b38002060 con 0x7f6b400b0770 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.203+0000 7f6b36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.203+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6b38003b10 con 0x7f6b400b0770 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.203+0000 7f6b36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.203+0000 7f6b36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.203+0000 7f6b36ffd640 10 monclient: dump: 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:15.191 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.203+0000 7f6b4486d640 4 set_mon_vals no callback set 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.204+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f6b38019b20 con 0x7f6b400b0770 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.204+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f6b38083b50 con 0x7f6b400b0770 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.204+0000 7f6b36ffd640 10 client.4515.objecter ms_dispatch 0x7f6b40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.204+0000 7f6b36ffd640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.204+0000 7f6b36ffd640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.204+0000 7f6b36ffd640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-29T10:47:15.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.206+0000 7f6b46070640 10 monclient: get_auth_request con 0x7f6b2406de90 auth_method 0 2024-03-29T10:47:15.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.210+0000 7f6b36ffd640 10 client.4515.objecter ms_handle_connect 0x7f6b2406de90 2024-03-29T10:47:15.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.212+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b3804f260 con 0x7f6b400b0770 2024-03-29T10:47:15.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.212+0000 7f6b36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:15.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.212+0000 7f6b36ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:15.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.537+0000 7f6b482d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:15.521 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.537+0000 7f6b482d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:15.521 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.538+0000 7f6b482d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:15.521 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.538+0000 7f6b482d4640 1 -- 172.21.15.114:0/1923185979 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b40075f10 con 0x7f6b400b0770 2024-03-29T10:47:15.521 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.538+0000 7f6b36ffd640 1 -- 172.21.15.114:0/1923185979 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f6b38051360 con 0x7f6b400b0770 2024-03-29T10:47:15.521 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.538+0000 7f6b36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:15.522 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.538+0000 7f6b36ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:15.522 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:47:15.522 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:15.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.542+0000 7f6b482d4640 10 librados: watch_flush enter 2024-03-29T10:47:15.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.542+0000 7f6b482d4640 10 librados: watch_flush exit 2024-03-29T10:47:15.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.542+0000 7f6b482d4640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-29T10:47:15.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.542+0000 7f6b482d4640 10 client.4515.objecter successfully canceled tick 2024-03-29T10:47:15.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.544+0000 7f6b482d4640 1 -- 172.21.15.114:0/1923185979 >> v1:172.21.15.114:6816/775044307 conn(0x7f6b2406de90 legacy=0x7f6b24070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:15.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.544+0000 7f6b482d4640 10 monclient: shutdown 2024-03-29T10:47:15.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.544+0000 7f6b482d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:15.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.544+0000 7f6b482d4640 1 -- 172.21.15.114:0/1923185979 >> v1:172.21.15.114:6789/0 conn(0x7f6b400b0770 legacy=0x7f6b40144df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:15.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.544+0000 7f6b482d4640 1 -- 172.21.15.114:0/1923185979 shutdown_connections 2024-03-29T10:47:15.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.544+0000 7f6b482d4640 1 -- 172.21.15.114:0/1923185979 >> 172.21.15.114:0/1923185979 conn(0x7f6b4006b660 msgr2=0x7f6b400b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:15.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.545+0000 7f6b482d4640 1 -- 172.21.15.114:0/1923185979 shutdown_connections 2024-03-29T10:47:15.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.545+0000 7f6b482d4640 1 -- 172.21.15.114:0/1923185979 wait complete. 2024-03-29T10:47:15.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:15.545+0000 7f6b482d4640 1 librados: shutdown 2024-03-29T10:47:15.547 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:22.354 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:22.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.517+0000 7f68a6616640 10 monclient: get_monmap_and_config 2024-03-29T10:47:22.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.517+0000 7f68a6616640 10 monclient: build_initial_monmap 2024-03-29T10:47:22.500 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.517+0000 7f68a6616640 10 monclient: monmap: 2024-03-29T10:47:22.500 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:22.500 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:22.500 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:47:22.518853+0000 2024-03-29T10:47:22.500 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:47:22.518853+0000 2024-03-29T10:47:22.500 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:22.501 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:22.501 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:22.501 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:47:22.501 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:47:22.501 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:22.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.518+0000 7f68a6616640 1 Processor -- start 2024-03-29T10:47:22.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.518+0000 7f68a6616640 1 -- start start 2024-03-29T10:47:22.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.518+0000 7f68a6616640 10 monclient: init 2024-03-29T10:47:22.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f68a6616640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:22.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f68a6616640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:22.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f68a6616640 10 monclient(hunting): picked mon.noname-c con 0x7f68a01457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:22.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f68a6616640 10 monclient(hunting): picked mon.noname-b con 0x7f68a013f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:22.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f689f7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f68a013f650 0x7f68a00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:53544/0 (socket says 172.21.15.114:53544) 2024-03-29T10:47:22.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f689f7fe640 1 -- 172.21.15.114:0/311627257 learned_addr learned my addr 172.21.15.114:0/311627257 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:22.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f68a6616640 10 monclient(hunting): picked mon.noname-a con 0x7f68a00a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:22.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f68a6616640 1 -- 172.21.15.114:0/311627257 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a00aaef0 con 0x7f68a00a9d70 2024-03-29T10:47:22.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f68a6616640 1 -- 172.21.15.114:0/311627257 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a0145bd0 con 0x7f68a01457f0 2024-03-29T10:47:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f68a6616640 1 -- 172.21.15.114:0/311627257 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a0145eb0 con 0x7f68a013f650 2024-03-29T10:47:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f68a6616640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.519+0000 7f68a6616640 10 monclient(hunting): authenticate will time out at 1307.760935s 2024-03-29T10:47:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.520+0000 7f689effd640 1 -- 172.21.15.114:0/311627257 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840154083 0 0) 0x7f68a0145eb0 con 0x7f68a013f650 2024-03-29T10:47:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.520+0000 7f689effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.520+0000 7f689effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.520+0000 7f689effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.520+0000 7f689effd640 10 monclient(hunting): my global_id is 4589 2024-03-29T10:47:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.520+0000 7f689effd640 1 -- 172.21.15.114:0/311627257 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6888003520 con 0x7f68a013f650 2024-03-29T10:47:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.521+0000 7f689effd640 1 -- 172.21.15.114:0/311627257 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2461514367 0 0) 0x7f6888003520 con 0x7f68a013f650 2024-03-29T10:47:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.521+0000 7f689effd640 1 -- 172.21.15.114:0/311627257 >> v1:172.21.15.176:6789/0 conn(0x7f68a01457f0 legacy=0x7f68a013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:22.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.523+0000 7f689effd640 1 -- 172.21.15.114:0/311627257 >> v1:172.21.15.114:6789/0 conn(0x7f68a00a9d70 legacy=0x7f68a014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:22.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.523+0000 7f689effd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:22.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.523+0000 7f689effd640 1 monclient: found mon.noname-b 2024-03-29T10:47:22.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.523+0000 7f689effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:22.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:47:22.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 1 -- 172.21.15.114:0/311627257 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a0146340 con 0x7f68a013f650 2024-03-29T10:47:22.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 10 monclient: _finish_auth 0 2024-03-29T10:47:22.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:22.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:22.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f68a6616640 5 monclient: authenticate success, global_id 4589 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f68a6616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 1 -- 172.21.15.114:0/311627257 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6890001690 con 0x7f68a013f650 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 10 monclient: dump: 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:22.509 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f68a6616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 1 -- 172.21.15.114:0/311627257 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6890001860 con 0x7f68a013f650 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f68a6616640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 1 -- 172.21.15.114:0/311627257 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6890001cb0 con 0x7f68a013f650 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f68a6616640 4 set_mon_vals no callback set 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.524+0000 7f689effd640 10 monclient: dump: 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:22.510 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:22.511 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:22.511 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:22.511 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:22.511 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:22.511 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:22.511 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:22.511 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:22.511 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:22.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.526+0000 7f68a6616640 10 monclient: shutdown 2024-03-29T10:47:22.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.526+0000 7f68a6616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:22.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.526+0000 7f68a6616640 1 -- 172.21.15.114:0/311627257 >> v1:172.21.15.114:6790/0 conn(0x7f68a013f650 legacy=0x7f68a00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:22.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.526+0000 7f68a6616640 1 -- 172.21.15.114:0/311627257 shutdown_connections 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.526+0000 7f68a6616640 1 -- 172.21.15.114:0/311627257 >> 172.21.15.114:0/311627257 conn(0x7f68a0066a00 msgr2=0x7f68a0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.527+0000 7f68a6616640 1 -- 172.21.15.114:0/311627257 shutdown_connections 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.527+0000 7f68a6616640 1 -- 172.21.15.114:0/311627257 wait complete. 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.528+0000 7f68a6616640 10 monclient: build_initial_monmap 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.528+0000 7f68a6616640 10 monclient: monmap: 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:47:22.514 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:47:22.515 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:22.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.528+0000 7f68a6616640 1 librados: starting msgr at 2024-03-29T10:47:22.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.528+0000 7f68a6616640 1 librados: starting objecter 2024-03-29T10:47:22.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.528+0000 7f68a6616640 1 Processor -- start 2024-03-29T10:47:22.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.529+0000 7f68a6616640 1 -- start start 2024-03-29T10:47:22.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.529+0000 7f68a6616640 1 librados: setting wanted keys 2024-03-29T10:47:22.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.529+0000 7f68a6616640 1 librados: calling monclient init 2024-03-29T10:47:22.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.529+0000 7f68a6616640 10 monclient: init 2024-03-29T10:47:22.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.529+0000 7f68a6616640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:22.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.529+0000 7f68a6616640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:22.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.530+0000 7f68a6616640 10 monclient(hunting): picked mon.noname-a con 0x7f68a00a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:22.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f68a6616640 10 monclient(hunting): picked mon.noname-b con 0x7f68a013f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:22.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f689f7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f68a013f650 0x7f68a01d7460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:56053/0 (socket says 172.21.15.114:56053) 2024-03-29T10:47:22.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f689f7fe640 1 -- 172.21.15.114:0/2188966466 learned_addr learned my addr 172.21.15.114:0/2188966466 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:22.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f689cff9640 10 client.?.objecter ms_handle_connect 0x7f68a013f650 2024-03-29T10:47:22.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f689cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:22.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f68a6616640 10 monclient(hunting): picked mon.noname-c con 0x7f68a01457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:22.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f689ffff640 1 --1- 172.21.15.114:0/2188966466 >> v1:172.21.15.114:6789/0 conn(0x7f68a00a9d70 0x7f68a01398d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37607/0 (socket says 172.21.15.114:37607) 2024-03-29T10:47:22.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a013a230 con 0x7f68a00a9d70 2024-03-29T10:47:22.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a013b2e0 con 0x7f68a013f650 2024-03-29T10:47:22.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a0138e60 con 0x7f68a01457f0 2024-03-29T10:47:22.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f68a6616640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:22.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.534+0000 7f68a6616640 10 monclient(hunting): authenticate will time out at 1307.775831s 2024-03-29T10:47:22.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 10 client.?.objecter ms_handle_connect 0x7f68a00a9d70 2024-03-29T10:47:22.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:22.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408354714 0 0) 0x7f68a013b2e0 con 0x7f68a013f650 2024-03-29T10:47:22.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:22.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:22.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:22.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 10 monclient(hunting): my global_id is 4489 2024-03-29T10:47:22.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f688c004eb0 con 0x7f68a013f650 2024-03-29T10:47:22.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782863823 0 0) 0x7f68a013a230 con 0x7f68a00a9d70 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 10 monclient(hunting): my global_id is 4518 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.535+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a013b2e0 con 0x7f68a00a9d70 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2193762970 0 0) 0x7f68a013b2e0 con 0x7f68a00a9d70 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68a013a230 con 0x7f68a00a9d70 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6894001790 con 0x7f68a00a9d70 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 10 monclient(hunting): dump: 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:22.527 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 246371742 0 0) 0x7f688c004eb0 con 0x7f68a013f650 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68a013b2e0 con 0x7f68a013f650 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6890001b70 con 0x7f68a013f650 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:22.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 10 monclient(hunting): dump: 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2031539103 0 0) 0x7f68a013a230 con 0x7f68a00a9d70 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 >> v1:172.21.15.114:6790/0 conn(0x7f68a01457f0 legacy=0x7f68a01d7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 >> v1:172.21.15.176:6789/0 conn(0x7f68a013f650 legacy=0x7f68a01d7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:22.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 1 monclient: found mon.a 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a013b9c0 con 0x7f68a00a9d70 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 10 monclient: _finish_auth 0 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.536+0000 7f689cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.537+0000 7f68a6616640 5 monclient: authenticate success, global_id 4518 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.537+0000 7f68a6616640 10 monclient: _renew_subs 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.537+0000 7f68a6616640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.537+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a013bbb0 con 0x7f68a00a9d70 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.537+0000 7f68a6616640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:22.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.537+0000 7f68a6616640 10 monclient: _renew_subs 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.537+0000 7f68a6616640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.538+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a01e2f50 con 0x7f68a00a9d70 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.538+0000 7f68a6616640 1 librados: init done 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.538+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6894001ca0 con 0x7f68a00a9d70 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.538+0000 7f689cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.538+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f6894004940 con 0x7f68a00a9d70 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.538+0000 7f689cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.539+0000 7f689cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.539+0000 7f689cff9640 10 monclient: dump: 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:22.531 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.539+0000 7f689dffb640 4 set_mon_vals no callback set 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.539+0000 7f68a6616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.539+0000 7f68a6616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.539+0000 7f68a6616640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.539+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6880005080 con 0x7f68a00a9d70 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.540+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f689401a910 con 0x7f68a00a9d70 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.541+0000 7f689f7fe640 10 monclient: get_auth_request con 0x7f688c06dd90 auth_method 0 2024-03-29T10:47:22.532 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.542+0000 7f689cff9640 10 client.4518.objecter ms_handle_connect 0x7f688c06dd90 2024-03-29T10:47:22.533 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.543+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f68940546b0 con 0x7f68a00a9d70 2024-03-29T10:47:22.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.543+0000 7f689cff9640 10 client.4518.objecter ms_dispatch 0x7f68a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:22.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.543+0000 7f689cff9640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:22.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.543+0000 7f689cff9640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:22.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.543+0000 7f689cff9640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-29T10:47:22.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.553+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6894050950 con 0x7f68a00a9d70 2024-03-29T10:47:22.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.553+0000 7f689cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:22.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.553+0000 7f689cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:22.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.879+0000 7f68a6616640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:22.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.879+0000 7f68a6616640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:22.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.879+0000 7f68a6616640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:22.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.879+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6880005330 con 0x7f68a00a9d70 2024-03-29T10:47:22.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.881+0000 7f689cff9640 1 -- 172.21.15.114:0/2188966466 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f6894050af0 con 0x7f68a00a9d70 2024-03-29T10:47:22.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.881+0000 7f689cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:22.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.881+0000 7f689cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:22.865 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:47:22.865 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:22.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.885+0000 7f68a6616640 10 librados: watch_flush enter 2024-03-29T10:47:22.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.886+0000 7f68a6616640 10 librados: watch_flush exit 2024-03-29T10:47:22.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.886+0000 7f68a6616640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-29T10:47:22.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.886+0000 7f68a6616640 10 client.4518.objecter successfully canceled tick 2024-03-29T10:47:22.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.887+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 >> v1:172.21.15.114:6816/775044307 conn(0x7f688c06dd90 legacy=0x7f688c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:22.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.887+0000 7f68a6616640 10 monclient: shutdown 2024-03-29T10:47:22.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.887+0000 7f68a6616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:22.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.887+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 >> v1:172.21.15.114:6789/0 conn(0x7f68a00a9d70 legacy=0x7f68a01398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:22.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.888+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 shutdown_connections 2024-03-29T10:47:22.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.888+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 >> 172.21.15.114:0/2188966466 conn(0x7f68a0066a00 msgr2=0x7f68a00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:22.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.888+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 shutdown_connections 2024-03-29T10:47:22.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.889+0000 7f68a6616640 1 -- 172.21.15.114:0/2188966466 wait complete. 2024-03-29T10:47:22.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:22.892+0000 7f68a6616640 1 librados: shutdown 2024-03-29T10:47:22.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:29.700 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:29.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.865+0000 7f9adfe82640 10 monclient: get_monmap_and_config 2024-03-29T10:47:29.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.865+0000 7f9adfe82640 10 monclient: build_initial_monmap 2024-03-29T10:47:29.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.865+0000 7f9adfe82640 10 monclient: monmap: 2024-03-29T10:47:29.847 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:29.847 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:29.847 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:47:29.865320+0000 2024-03-29T10:47:29.847 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:47:29.865320+0000 2024-03-29T10:47:29.847 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:29.847 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:29.847 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:29.847 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:47:29.848 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:47:29.848 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:29.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.866+0000 7f9adfe82640 1 Processor -- start 2024-03-29T10:47:29.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 1 -- start start 2024-03-29T10:47:29.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 10 monclient: init 2024-03-29T10:47:29.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:29.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:29.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad80b4790 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:29.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad80aea90 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:29.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad80b39f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:29.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad80b0c60 con 0x7f9ad80b39f0 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad80af020 con 0x7f9ad80b4790 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad813fbd0 con 0x7f9ad80aea90 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.867+0000 7f9adfe82640 10 monclient(hunting): authenticate will time out at 1315.107935s 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.868+0000 7f9add41d640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f9ad80aea90 0x7f9ad80af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:53734/0 (socket says 172.21.15.114:53734) 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.868+0000 7f9add41d640 1 -- 172.21.15.114:0/1907863735 learned_addr learned my addr 172.21.15.114:0/1907863735 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.869+0000 7f9adcc1c640 1 -- 172.21.15.114:0/1907863735 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593542016 0 0) 0x7f9ad813fbd0 con 0x7f9ad80aea90 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.869+0000 7f9adcc1c640 10 monclient(hunting): _negotiate 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.869+0000 7f9adcc1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.869+0000 7f9adcc1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.869+0000 7f9adcc1c640 10 monclient(hunting): my global_id is 4592 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.869+0000 7f9adcc1c640 1 -- 172.21.15.114:0/1907863735 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac4003520 con 0x7f9ad80aea90 2024-03-29T10:47:29.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.869+0000 7f9adcc1c640 1 -- 172.21.15.114:0/1907863735 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3442351096 0 0) 0x7f9ac4003520 con 0x7f9ad80aea90 2024-03-29T10:47:29.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.869+0000 7f9adcc1c640 1 -- 172.21.15.114:0/1907863735 >> v1:172.21.15.176:6789/0 conn(0x7f9ad80b4790 legacy=0x7f9ad80b1080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:29.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 1 -- 172.21.15.114:0/1907863735 >> v1:172.21.15.114:6789/0 conn(0x7f9ad80b39f0 legacy=0x7f9ad813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:29.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 10 monclient: _finish_hunting 0 2024-03-29T10:47:29.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 1 monclient: found mon.noname-b 2024-03-29T10:47:29.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:29.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:47:29.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 1 -- 172.21.15.114:0/1907863735 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad813fd60 con 0x7f9ad80aea90 2024-03-29T10:47:29.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 10 monclient: _finish_auth 0 2024-03-29T10:47:29.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 10 monclient: _check_auth_tickets 2024-03-29T10:47:29.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:29.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 1 -- 172.21.15.114:0/1907863735 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9ad4001690 con 0x7f9ad80aea90 2024-03-29T10:47:29.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:29.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adcc1c640 10 monclient: dump: 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adfe82640 5 monclient: authenticate success, global_id 4592 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.871+0000 7f9adfe82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:29.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.872+0000 7f9adcc1c640 1 -- 172.21.15.114:0/1907863735 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ad4001860 con 0x7f9ad80aea90 2024-03-29T10:47:29.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.872+0000 7f9adcc1c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:29.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.872+0000 7f9adfe82640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:29.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.872+0000 7f9adfe82640 4 set_mon_vals no callback set 2024-03-29T10:47:29.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.873+0000 7f9adfe82640 10 monclient: shutdown 2024-03-29T10:47:29.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.873+0000 7f9adfe82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:29.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.873+0000 7f9adfe82640 1 -- 172.21.15.114:0/1907863735 >> v1:172.21.15.114:6790/0 conn(0x7f9ad80aea90 legacy=0x7f9ad80af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:29.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.873+0000 7f9adcc1c640 1 -- 172.21.15.114:0/1907863735 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9ad4001cb0 con 0x7f9ad80aea90 2024-03-29T10:47:29.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.873+0000 7f9adcc1c640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:47:29.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.873+0000 7f9adfe82640 1 -- 172.21.15.114:0/1907863735 shutdown_connections 2024-03-29T10:47:29.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.873+0000 7f9adfe82640 1 -- 172.21.15.114:0/1907863735 >> 172.21.15.114:0/1907863735 conn(0x7f9ad8066a00 msgr2=0x7f9ad8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:29.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.873+0000 7f9adfe82640 1 -- 172.21.15.114:0/1907863735 shutdown_connections 2024-03-29T10:47:29.859 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.873+0000 7f9adfe82640 1 -- 172.21.15.114:0/1907863735 wait complete. 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.874+0000 7f9adfe82640 10 monclient: build_initial_monmap 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.874+0000 7f9adfe82640 10 monclient: monmap: 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.875+0000 7f9adfe82640 1 librados: starting msgr at 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.875+0000 7f9adfe82640 1 librados: starting objecter 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.875+0000 7f9adfe82640 1 Processor -- start 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.875+0000 7f9adfe82640 1 -- start start 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.875+0000 7f9adfe82640 1 librados: setting wanted keys 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.875+0000 7f9adfe82640 1 librados: calling monclient init 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.875+0000 7f9adfe82640 10 monclient: init 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.876+0000 7f9adfe82640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.876+0000 7f9adfe82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.876+0000 7f9adfe82640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad80b39f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.876+0000 7f9adfe82640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad80b4790 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.876+0000 7f9adfe82640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad812ad10 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.876+0000 7f9adfe82640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad8135610 con 0x7f9ad80b39f0 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.876+0000 7f9adfe82640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad8141540 con 0x7f9ad812ad10 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.876+0000 7f9add41d640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f9ad80b4790 0x7f9ad812a600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:56226/0 (socket says 172.21.15.114:56226) 2024-03-29T10:47:29.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.876+0000 7f9add41d640 1 -- 172.21.15.114:0/273032209 learned_addr learned my addr 172.21.15.114:0/273032209 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:29.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.878+0000 7f9adfe82640 1 -- 172.21.15.114:0/273032209 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad8141820 con 0x7f9ad80b4790 2024-03-29T10:47:29.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.878+0000 7f9adfe82640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:29.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.878+0000 7f9adfe82640 10 monclient(hunting): authenticate will time out at 1315.118910s 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 10 client.?.objecter ms_handle_connect 0x7f9ad80b4790 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 10 client.?.objecter ms_handle_connect 0x7f9ad80b39f0 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 10 client.?.objecter ms_handle_connect 0x7f9ad812ad10 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028977785 0 0) 0x7f9ad8141820 con 0x7f9ad80b4790 2024-03-29T10:47:29.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:29.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:29.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:29.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 10 monclient(hunting): my global_id is 4595 2024-03-29T10:47:29.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.879+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9abc004eb0 con 0x7f9ad80b4790 2024-03-29T10:47:29.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602001216 0 0) 0x7f9ad8141540 con 0x7f9ad812ad10 2024-03-29T10:47:29.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad8141820 con 0x7f9ad812ad10 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1647545351 0 0) 0x7f9abc004eb0 con 0x7f9ad80b4790 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ad8141540 con 0x7f9ad80b4790 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9ad4001bb0 con 0x7f9ad80b4790 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient(hunting): dump: 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:29.868 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:29.869 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:29.869 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:29.869 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:29.869 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:29.869 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:29.869 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:29.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:29.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 855180190 0 0) 0x7f9ad8141820 con 0x7f9ad812ad10 2024-03-29T10:47:29.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9abc004eb0 con 0x7f9ad812ad10 2024-03-29T10:47:29.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1081880832 0 0) 0x7f9ad8141540 con 0x7f9ad80b4790 2024-03-29T10:47:29.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 >> v1:172.21.15.176:6789/0 conn(0x7f9ad812ad10 legacy=0x7f9ad81e5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 >> v1:172.21.15.114:6789/0 conn(0x7f9ad80b39f0 legacy=0x7f9ad81401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 monclient: found mon.c 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad8141b00 con 0x7f9ad80b4790 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ad4001d10 con 0x7f9ad80b4790 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9ace7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.880+0000 7f9acffff640 4 set_mon_vals no callback set 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.881+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9ad4004e90 con 0x7f9ad80b4790 2024-03-29T10:47:29.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.881+0000 7f9ace7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.881+0000 7f9ace7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.881+0000 7f9ace7fc640 10 monclient: dump: 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:29.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.882+0000 7f9adfe82640 5 monclient: authenticate success, global_id 4595 2024-03-29T10:47:29.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.882+0000 7f9adfe82640 10 monclient: _renew_subs 2024-03-29T10:47:29.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.882+0000 7f9adfe82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:29.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.882+0000 7f9adfe82640 1 -- 172.21.15.114:0/273032209 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad8140990 con 0x7f9ad80b4790 2024-03-29T10:47:29.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.882+0000 7f9adfe82640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:29.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.882+0000 7f9adfe82640 10 monclient: _renew_subs 2024-03-29T10:47:29.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.882+0000 7f9adfe82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:29.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.882+0000 7f9adfe82640 1 -- 172.21.15.114:0/273032209 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad81293f0 con 0x7f9ad80b4790 2024-03-29T10:47:29.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.882+0000 7f9adfe82640 1 librados: init done 2024-03-29T10:47:29.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.884+0000 7f9adfe82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:29.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.884+0000 7f9adfe82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:29.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.884+0000 7f9adfe82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:29.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.884+0000 7f9adfe82640 1 -- 172.21.15.114:0/273032209 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad8075f10 con 0x7f9ad80b4790 2024-03-29T10:47:29.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.886+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f9ad401ae60 con 0x7f9ad80b4790 2024-03-29T10:47:29.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.887+0000 7f9addc1e640 10 monclient: get_auth_request con 0x7f9abc07ef80 auth_method 0 2024-03-29T10:47:29.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.887+0000 7f9ace7fc640 10 client.4595.objecter ms_handle_connect 0x7f9abc07ef80 2024-03-29T10:47:29.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.887+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f9ad4084c20 con 0x7f9ad80b4790 2024-03-29T10:47:29.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.887+0000 7f9ace7fc640 10 client.4595.objecter ms_dispatch 0x7f9ad8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:29.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.887+0000 7f9ace7fc640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:29.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.887+0000 7f9ace7fc640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:29.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.887+0000 7f9ace7fc640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T10:47:29.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.895+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ad4050500 con 0x7f9ad80b4790 2024-03-29T10:47:29.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.895+0000 7f9ace7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:29.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:29.895+0000 7f9ace7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:30.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.233+0000 7f9adfe82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:30.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.233+0000 7f9adfe82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:30.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.233+0000 7f9adfe82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:30.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.233+0000 7f9adfe82640 1 -- 172.21.15.114:0/273032209 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ad8074ed0 con 0x7f9ad80b4790 2024-03-29T10:47:30.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.234+0000 7f9ace7fc640 1 -- 172.21.15.114:0/273032209 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f9ad4052430 con 0x7f9ad80b4790 2024-03-29T10:47:30.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.234+0000 7f9ace7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:30.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.234+0000 7f9ace7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:30.216 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:47:30.216 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:30.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.237+0000 7f9adfe82640 10 librados: watch_flush enter 2024-03-29T10:47:30.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 10 librados: watch_flush exit 2024-03-29T10:47:30.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T10:47:30.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 10 client.4595.objecter successfully canceled tick 2024-03-29T10:47:30.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 1 -- 172.21.15.114:0/273032209 >> v1:172.21.15.114:6816/775044307 conn(0x7f9abc07ef80 legacy=0x7f9abc081440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:30.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 10 monclient: shutdown 2024-03-29T10:47:30.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:30.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 1 -- 172.21.15.114:0/273032209 >> v1:172.21.15.114:6790/0 conn(0x7f9ad80b4790 legacy=0x7f9ad812a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:30.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 1 -- 172.21.15.114:0/273032209 shutdown_connections 2024-03-29T10:47:30.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 1 -- 172.21.15.114:0/273032209 >> 172.21.15.114:0/273032209 conn(0x7f9ad8066a00 msgr2=0x7f9ad80b24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:30.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 1 -- 172.21.15.114:0/273032209 shutdown_connections 2024-03-29T10:47:30.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 1 -- 172.21.15.114:0/273032209 wait complete. 2024-03-29T10:47:30.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:30.238+0000 7f9adfe82640 1 librados: shutdown 2024-03-29T10:47:30.237 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:37.045 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:37.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.241+0000 7fe713a43640 10 monclient: get_monmap_and_config 2024-03-29T10:47:37.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.241+0000 7fe713a43640 10 monclient: build_initial_monmap 2024-03-29T10:47:37.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.241+0000 7fe713a43640 10 monclient: monmap: 2024-03-29T10:47:37.223 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:37.223 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:37.223 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:47:37.241342+0000 2024-03-29T10:47:37.224 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:47:37.241342+0000 2024-03-29T10:47:37.224 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:37.224 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:37.224 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:37.224 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:47:37.224 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:47:37.224 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:37.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.244+0000 7fe713a43640 1 Processor -- start 2024-03-29T10:47:37.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.244+0000 7fe713a43640 1 -- start start 2024-03-29T10:47:37.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.244+0000 7fe713a43640 10 monclient: init 2024-03-29T10:47:37.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.245+0000 7fe713a43640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:37.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.245+0000 7fe713a43640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:37.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.245+0000 7fe713a43640 10 monclient(hunting): picked mon.noname-c con 0x7fe70c0b0770 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:37.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.245+0000 7fe713a43640 10 monclient(hunting): picked mon.noname-b con 0x7fe70c14a700 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:37.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.245+0000 7fe710fde640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fe70c14a700 0x7fe70c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:46388/0 (socket says 172.21.15.114:46388) 2024-03-29T10:47:37.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.245+0000 7fe710fde640 1 -- 172.21.15.114:0/1618328367 learned_addr learned my addr 172.21.15.114:0/1618328367 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:37.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.246+0000 7fe713a43640 10 monclient(hunting): picked mon.noname-a con 0x7fe70c0ac5a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:37.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.246+0000 7fe713a43640 1 -- 172.21.15.114:0/1618328367 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70c0aba70 con 0x7fe70c0ac5a0 2024-03-29T10:47:37.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.246+0000 7fe713a43640 1 -- 172.21.15.114:0/1618328367 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70c0aecb0 con 0x7fe70c0b0770 2024-03-29T10:47:37.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.246+0000 7fe713a43640 1 -- 172.21.15.114:0/1618328367 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70c0af150 con 0x7fe70c14a700 2024-03-29T10:47:37.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.246+0000 7fe713a43640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:37.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.246+0000 7fe713a43640 10 monclient(hunting): authenticate will time out at 1322.486351s 2024-03-29T10:47:37.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 1 -- 172.21.15.114:0/1618328367 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988053112 0 0) 0x7fe70c0af150 con 0x7fe70c14a700 2024-03-29T10:47:37.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:37.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:37.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:37.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 10 monclient(hunting): my global_id is 4598 2024-03-29T10:47:37.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 1 -- 172.21.15.114:0/1618328367 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f8003520 con 0x7fe70c14a700 2024-03-29T10:47:37.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 1 -- 172.21.15.114:0/1618328367 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21671748 0 0) 0x7fe70c0aecb0 con 0x7fe70c0b0770 2024-03-29T10:47:37.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:37.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:37.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:37.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 10 monclient(hunting): my global_id is 4495 2024-03-29T10:47:37.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.247+0000 7fe703fff640 1 -- 172.21.15.114:0/1618328367 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe70c0af150 con 0x7fe70c0b0770 2024-03-29T10:47:37.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.248+0000 7fe703fff640 1 -- 172.21.15.114:0/1618328367 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3603003654 0 0) 0x7fe70c0af150 con 0x7fe70c0b0770 2024-03-29T10:47:37.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.248+0000 7fe703fff640 1 -- 172.21.15.114:0/1618328367 >> v1:172.21.15.114:6790/0 conn(0x7fe70c14a700 legacy=0x7fe70c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:37.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.248+0000 7fe703fff640 1 -- 172.21.15.114:0/1618328367 >> v1:172.21.15.114:6789/0 conn(0x7fe70c0ac5a0 legacy=0x7fe70c151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:37.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.249+0000 7fe703fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:37.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.249+0000 7fe703fff640 1 monclient: found mon.noname-c 2024-03-29T10:47:37.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.249+0000 7fe703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:37.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.249+0000 7fe703fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:47:37.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.249+0000 7fe703fff640 1 -- 172.21.15.114:0/1618328367 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c0ac9d0 con 0x7fe70c0b0770 2024-03-29T10:47:37.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.249+0000 7fe703fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:37.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.249+0000 7fe703fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:37.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.249+0000 7fe703fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.249+0000 7fe703fff640 1 -- 172.21.15.114:0/1618328367 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe708001520 con 0x7fe70c0b0770 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.249+0000 7fe713a43640 5 monclient: authenticate success, global_id 4495 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.249+0000 7fe713a43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.250+0000 7fe703fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.250+0000 7fe703fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.250+0000 7fe703fff640 10 monclient: dump: 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:37.235 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:37.235 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:37.235 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:37.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.250+0000 7fe713a43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:37.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.250+0000 7fe703fff640 1 -- 172.21.15.114:0/1618328367 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe708001680 con 0x7fe70c0b0770 2024-03-29T10:47:37.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.250+0000 7fe703fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:37.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.250+0000 7fe703fff640 1 -- 172.21.15.114:0/1618328367 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe708001a50 con 0x7fe70c0b0770 2024-03-29T10:47:37.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.250+0000 7fe703fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:37.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.250+0000 7fe703fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:37.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.250+0000 7fe703fff640 10 monclient: dump: 2024-03-29T10:47:37.235 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:37.236 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:37.236 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:37.236 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:37.236 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:37.236 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:37.236 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:37.236 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:37.236 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:37.236 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:37.236 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.250+0000 7fe713a43640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:37.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.251+0000 7fe713a43640 4 set_mon_vals no callback set 2024-03-29T10:47:37.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.252+0000 7fe713a43640 10 monclient: shutdown 2024-03-29T10:47:37.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.252+0000 7fe713a43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:37.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.252+0000 7fe713a43640 1 -- 172.21.15.114:0/1618328367 >> v1:172.21.15.176:6789/0 conn(0x7fe70c0b0770 legacy=0x7fe70c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:37.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.252+0000 7fe713a43640 1 -- 172.21.15.114:0/1618328367 shutdown_connections 2024-03-29T10:47:37.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.253+0000 7fe713a43640 1 -- 172.21.15.114:0/1618328367 >> 172.21.15.114:0/1618328367 conn(0x7fe70c06b660 msgr2=0x7fe70c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:37.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.253+0000 7fe713a43640 1 -- 172.21.15.114:0/1618328367 shutdown_connections 2024-03-29T10:47:37.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.253+0000 7fe713a43640 1 -- 172.21.15.114:0/1618328367 wait complete. 2024-03-29T10:47:37.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.254+0000 7fe713a43640 10 monclient: build_initial_monmap 2024-03-29T10:47:37.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.254+0000 7fe713a43640 10 monclient: monmap: 2024-03-29T10:47:37.240 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:37.240 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:37.240 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:47:37.240 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:47:37.240 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:37.240 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:37.240 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:37.240 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:47:37.240 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:47:37.240 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:37.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.255+0000 7fe713a43640 1 librados: starting msgr at 2024-03-29T10:47:37.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.255+0000 7fe713a43640 1 librados: starting objecter 2024-03-29T10:47:37.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.255+0000 7fe713a43640 1 Processor -- start 2024-03-29T10:47:37.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.255+0000 7fe713a43640 1 -- start start 2024-03-29T10:47:37.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.255+0000 7fe713a43640 1 librados: setting wanted keys 2024-03-29T10:47:37.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.255+0000 7fe713a43640 1 librados: calling monclient init 2024-03-29T10:47:37.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.255+0000 7fe713a43640 10 monclient: init 2024-03-29T10:47:37.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.256+0000 7fe713a43640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:37.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.256+0000 7fe713a43640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:37.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.258+0000 7fe713a43640 10 monclient(hunting): picked mon.noname-a con 0x7fe70c0ac5a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:37.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.258+0000 7fe7117df640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe70c0ac5a0 0x7fe70c1decf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59959/0 (socket says 172.21.15.114:59959) 2024-03-29T10:47:37.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.258+0000 7fe713a43640 10 monclient(hunting): picked mon.noname-b con 0x7fe70c0b0770 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:37.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.258+0000 7fe713a43640 10 monclient(hunting): picked mon.noname-c con 0x7fe70c14a700 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:37.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.258+0000 7fe713a43640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70c125e80 con 0x7fe70c0ac5a0 2024-03-29T10:47:37.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.258+0000 7fe713a43640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70c1264d0 con 0x7fe70c0b0770 2024-03-29T10:47:37.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.258+0000 7fe713a43640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70c1239d0 con 0x7fe70c14a700 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.258+0000 7fe713a43640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.258+0000 7fe713a43640 10 monclient(hunting): authenticate will time out at 1322.498587s 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.258+0000 7fe7117df640 1 -- 172.21.15.114:0/3453891921 learned_addr learned my addr 172.21.15.114:0/3453891921 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 client.?.objecter ms_handle_connect 0x7fe70c0ac5a0 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 client.?.objecter ms_handle_connect 0x7fe70c0b0770 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 client.?.objecter ms_handle_connect 0x7fe70c14a700 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212817235 0 0) 0x7fe70c125e80 con 0x7fe70c0ac5a0 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:37.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): my global_id is 4524 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f0004eb0 con 0x7fe70c0ac5a0 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924215890 0 0) 0x7fe70c1264d0 con 0x7fe70c0b0770 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe70c125e80 con 0x7fe70c0b0770 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558425493 0 0) 0x7fe70c1239d0 con 0x7fe70c14a700 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:37.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 10 monclient(hunting): my global_id is 4601 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.259+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe70c1264d0 con 0x7fe70c14a700 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3680241428 0 0) 0x7fe70c125e80 con 0x7fe70c0b0770 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe70c1239d0 con 0x7fe70c0b0770 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 61310398 0 0) 0x7fe6f0004eb0 con 0x7fe70c0ac5a0 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe70c125e80 con 0x7fe70c0ac5a0 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe6fc0017d0 con 0x7fe70c0b0770 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:37.252 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3543288763 0 0) 0x7fe70c1264d0 con 0x7fe70c14a700 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6f0004eb0 con 0x7fe70c14a700 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe708001990 con 0x7fe70c0ac5a0 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:37.253 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 890550995 0 0) 0x7fe70c1239d0 con 0x7fe70c0b0770 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 >> v1:172.21.15.114:6790/0 conn(0x7fe70c14a700 legacy=0x7fe70c144df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 >> v1:172.21.15.114:6789/0 conn(0x7fe70c0ac5a0 legacy=0x7fe70c1decf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 1 monclient: found mon.b 2024-03-29T10:47:37.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.260+0000 7fe701ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.261+0000 7fe701ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.261+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c126110 con 0x7fe70c0b0770 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.261+0000 7fe701ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.261+0000 7fe701ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.261+0000 7fe701ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.261+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6fc001ce0 con 0x7fe70c0b0770 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.261+0000 7fe701ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.261+0000 7fe7037fe640 4 set_mon_vals no callback set 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fe6fc004980 con 0x7fe70c0b0770 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe701ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe701ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe701ffb640 10 monclient: dump: 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:37.255 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:37.256 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:37.256 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:37.256 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:37.256 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:37.256 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:37.256 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:37.256 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:37.256 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:37.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe713a43640 5 monclient: authenticate success, global_id 4498 2024-03-29T10:47:37.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe713a43640 10 monclient: _renew_subs 2024-03-29T10:47:37.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe713a43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:47:37.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe713a43640 1 -- 172.21.15.114:0/3453891921 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70c1df980 con 0x7fe70c0b0770 2024-03-29T10:47:37.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe713a43640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:37.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe713a43640 10 monclient: _renew_subs 2024-03-29T10:47:37.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe713a43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:47:37.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe713a43640 1 -- 172.21.15.114:0/3453891921 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70c1272e0 con 0x7fe70c0b0770 2024-03-29T10:47:37.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.262+0000 7fe713a43640 1 librados: init done 2024-03-29T10:47:37.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.264+0000 7fe6ef7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:37.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.265+0000 7fe6ef7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:37.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.265+0000 7fe6ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:47:37.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.265+0000 7fe6ef7fe640 1 -- 172.21.15.114:0/3453891921 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70c0ab560 con 0x7fe70c0b0770 2024-03-29T10:47:37.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.265+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fe6fc01a950 con 0x7fe70c0b0770 2024-03-29T10:47:37.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.266+0000 7fe7117df640 10 monclient: get_auth_request con 0x7fe6f006e180 auth_method 0 2024-03-29T10:47:37.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.266+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fe6fc0848d0 con 0x7fe70c0b0770 2024-03-29T10:47:37.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.266+0000 7fe701ffb640 10 client.4498.objecter ms_dispatch 0x7fe70c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:37.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.266+0000 7fe701ffb640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:37.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.266+0000 7fe701ffb640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:37.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.266+0000 7fe701ffb640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-29T10:47:37.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.266+0000 7fe701ffb640 10 client.4498.objecter ms_handle_connect 0x7fe6f006e180 2024-03-29T10:47:37.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.272+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6fc04ffe0 con 0x7fe70c0b0770 2024-03-29T10:47:37.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.272+0000 7fe701ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:37.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.272+0000 7fe701ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:37.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.604+0000 7fe6ef7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:37.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.604+0000 7fe6ef7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:37.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.604+0000 7fe6ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:47:37.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.604+0000 7fe6ef7fe640 1 -- 172.21.15.114:0/3453891921 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe70c1dfc80 con 0x7fe70c0b0770 2024-03-29T10:47:37.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.605+0000 7fe701ffb640 1 -- 172.21.15.114:0/3453891921 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fe70c1dfc80 con 0x7fe70c0b0770 2024-03-29T10:47:37.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.605+0000 7fe701ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:37.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.605+0000 7fe701ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:37.587 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:47:37.588 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:37.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.609+0000 7fe6ef7fe640 10 librados: watch_flush enter 2024-03-29T10:47:37.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.609+0000 7fe6ef7fe640 10 librados: watch_flush exit 2024-03-29T10:47:37.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.609+0000 7fe6ef7fe640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-29T10:47:37.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.609+0000 7fe6ef7fe640 10 client.4498.objecter successfully canceled tick 2024-03-29T10:47:37.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.609+0000 7fe6ef7fe640 1 -- 172.21.15.114:0/3453891921 >> v1:172.21.15.114:6816/775044307 conn(0x7fe6f006e180 legacy=0x7fe6f0070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:37.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.609+0000 7fe6ef7fe640 10 monclient: shutdown 2024-03-29T10:47:37.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.609+0000 7fe6ef7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:37.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.609+0000 7fe6ef7fe640 1 -- 172.21.15.114:0/3453891921 >> v1:172.21.15.176:6789/0 conn(0x7fe70c0b0770 legacy=0x7fe70c1269f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:37.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.609+0000 7fe6ef7fe640 1 -- 172.21.15.114:0/3453891921 shutdown_connections 2024-03-29T10:47:37.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.609+0000 7fe6ef7fe640 1 -- 172.21.15.114:0/3453891921 >> 172.21.15.114:0/3453891921 conn(0x7fe70c06b660 msgr2=0x7fe70c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:37.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.609+0000 7fe6ef7fe640 1 -- 172.21.15.114:0/3453891921 shutdown_connections 2024-03-29T10:47:37.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.610+0000 7fe6ef7fe640 1 -- 172.21.15.114:0/3453891921 wait complete. 2024-03-29T10:47:37.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:37.610+0000 7fe6ef7fe640 1 librados: shutdown 2024-03-29T10:47:37.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:44.416 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:44.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.578+0000 7fb4f3fff640 10 monclient: get_monmap_and_config 2024-03-29T10:47:44.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.578+0000 7fb4f3fff640 10 monclient: build_initial_monmap 2024-03-29T10:47:44.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.578+0000 7fb4f3fff640 10 monclient: monmap: 2024-03-29T10:47:44.561 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:44.561 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:44.561 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:47:44.579805+0000 2024-03-29T10:47:44.561 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:47:44.579805+0000 2024-03-29T10:47:44.561 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:44.561 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:44.561 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:44.562 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:47:44.562 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:47:44.562 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:44.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.579+0000 7fb4f3fff640 1 Processor -- start 2024-03-29T10:47:44.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.579+0000 7fb4f3fff640 1 -- start start 2024-03-29T10:47:44.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.579+0000 7fb4f3fff640 10 monclient: init 2024-03-29T10:47:44.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.580+0000 7fb4f3fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:44.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.580+0000 7fb4f3fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:44.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.580+0000 7fb4f3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb4f40b4820 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:44.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.580+0000 7fb4f3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb4f40a9ce0 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:44.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.580+0000 7fb4f3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb4f40b3cd0 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:44.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.580+0000 7fb4f3fff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f40b4180 con 0x7fb4f40b4820 2024-03-29T10:47:44.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.580+0000 7fb4f3fff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f40aaf60 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.580+0000 7fb4f3fff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f413c200 con 0x7fb4f40b3cd0 2024-03-29T10:47:44.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.580+0000 7fb4f3fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:44.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.580+0000 7fb4f3fff640 10 monclient(hunting): authenticate will time out at 1329.821818s 2024-03-29T10:47:44.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.581+0000 7fb4f2ffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb4f40b4820 0x7fb4f413baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49606/0 (socket says 172.21.15.114:49606) 2024-03-29T10:47:44.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.581+0000 7fb4f2ffd640 1 -- 172.21.15.114:0/4176048891 learned_addr learned my addr 172.21.15.114:0/4176048891 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.581+0000 7fb4f1ffb640 1 -- 172.21.15.114:0/4176048891 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838089622 0 0) 0x7fb4ec001490 con 0x7fb4f40b4820 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.581+0000 7fb4f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.581+0000 7fb4f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.581+0000 7fb4f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.581+0000 7fb4f1ffb640 10 monclient(hunting): my global_id is 4527 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.581+0000 7fb4f1ffb640 1 -- 172.21.15.114:0/4176048891 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e0003520 con 0x7fb4f40b4820 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 1 -- 172.21.15.114:0/4176048891 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656490287 0 0) 0x7fb4f413c200 con 0x7fb4f40b3cd0 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient(hunting): my global_id is 4604 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 1 -- 172.21.15.114:0/4176048891 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4ec001490 con 0x7fb4f40b3cd0 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 1 -- 172.21.15.114:0/4176048891 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 94247571 0 0) 0x7fb4e0003520 con 0x7fb4f40b4820 2024-03-29T10:47:44.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 1 -- 172.21.15.114:0/4176048891 >> v1:172.21.15.114:6790/0 conn(0x7fb4f40b3cd0 legacy=0x7fb4f414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 1 -- 172.21.15.114:0/4176048891 >> v1:172.21.15.176:6789/0 conn(0x7fb4f40a9ce0 legacy=0x7fb4f4145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 1 monclient: found mon.noname-a 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 1 -- 172.21.15.114:0/4176048891 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f4145e80 con 0x7fb4f40b4820 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f3fff640 5 monclient: authenticate success, global_id 4527 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 1 -- 172.21.15.114:0/4176048891 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb4ec001980 con 0x7fb4f40b4820 2024-03-29T10:47:44.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient: dump: 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 1 -- 172.21.15.114:0/4176048891 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4ec001ba0 con 0x7fb4f40b4820 2024-03-29T10:47:44.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f3fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f3fff640 4 set_mon_vals no callback set 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 1 -- 172.21.15.114:0/4176048891 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb4ec001f30 con 0x7fb4f40b4820 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.582+0000 7fb4f1ffb640 10 monclient: dump: 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:44.568 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:44.569 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:44.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.583+0000 7fb4f3fff640 10 monclient: shutdown 2024-03-29T10:47:44.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.583+0000 7fb4f3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:44.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.583+0000 7fb4f3fff640 1 -- 172.21.15.114:0/4176048891 >> v1:172.21.15.114:6789/0 conn(0x7fb4f40b4820 legacy=0x7fb4f413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.583+0000 7fb4f3fff640 1 -- 172.21.15.114:0/4176048891 shutdown_connections 2024-03-29T10:47:44.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.583+0000 7fb4f3fff640 1 -- 172.21.15.114:0/4176048891 >> 172.21.15.114:0/4176048891 conn(0x7fb4f406d070 msgr2=0x7fb4f4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:44.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.583+0000 7fb4f3fff640 1 -- 172.21.15.114:0/4176048891 shutdown_connections 2024-03-29T10:47:44.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.583+0000 7fb4f3fff640 1 -- 172.21.15.114:0/4176048891 wait complete. 2024-03-29T10:47:44.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.584+0000 7fb4f3fff640 10 monclient: build_initial_monmap 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.584+0000 7fb4f3fff640 10 monclient: monmap: 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.584+0000 7fb4f3fff640 1 librados: starting msgr at 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.584+0000 7fb4f3fff640 1 librados: starting objecter 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.585+0000 7fb4f3fff640 1 Processor -- start 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.585+0000 7fb4f3fff640 1 -- start start 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.585+0000 7fb4f3fff640 1 librados: setting wanted keys 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.585+0000 7fb4f3fff640 1 librados: calling monclient init 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.585+0000 7fb4f3fff640 10 monclient: init 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f3fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f3fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb4f40a9ce0 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb4f40b3cd0 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb4f40b4820 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f3fff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f41df0c0 con 0x7fb4f40b3cd0 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f3fff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f40ad3c0 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f3fff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f413d6f0 con 0x7fb4f40b4820 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f3fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f3fff640 10 monclient(hunting): authenticate will time out at 1329.827295s 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f27fc640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb4f40b3cd0 0x7fb4f413c8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49990/0 (socket says 172.21.15.114:49990) 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4f27fc640 1 -- 172.21.15.114:0/1640344457 learned_addr learned my addr 172.21.15.114:0/1640344457 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4ebfff640 10 client.?.objecter ms_handle_connect 0x7fb4f40b3cd0 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4ebfff640 10 client.?.objecter ms_handle_connect 0x7fb4f40a9ce0 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297141934 0 0) 0x7fb4f41df0c0 con 0x7fb4f40b3cd0 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4ebfff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.586+0000 7fb4ebfff640 10 monclient(hunting): my global_id is 4530 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.587+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d4004eb0 con 0x7fb4f40b3cd0 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.587+0000 7fb4ebfff640 10 client.?.objecter ms_handle_connect 0x7fb4f40b4820 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.587+0000 7fb4ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.587+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905809524 0 0) 0x7fb4f40ad3c0 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.587+0000 7fb4ebfff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.587+0000 7fb4ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.587+0000 7fb4ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.587+0000 7fb4ebfff640 10 monclient(hunting): my global_id is 4504 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.587+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4f41df0c0 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597678203 0 0) 0x7fb4f413d6f0 con 0x7fb4f40b4820 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): my global_id is 4607 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4f40ad3c0 con 0x7fb4f40b4820 2024-03-29T10:47:44.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3370087391 0 0) 0x7fb4d4004eb0 con 0x7fb4f40b3cd0 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4f413d6f0 con 0x7fb4f40b3cd0 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb4dc001960 con 0x7fb4f40b3cd0 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): dump: 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:44.583 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 800423575 0 0) 0x7fb4f41df0c0 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4d4004eb0 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb4ec001eb0 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): dump: 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668386502 0 0) 0x7fb4f40ad3c0 con 0x7fb4f40b4820 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4f41df0c0 con 0x7fb4f40b4820 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb4e40018f0 con 0x7fb4f40b4820 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient(hunting): dump: 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 66083049 0 0) 0x7fb4d4004eb0 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 >> v1:172.21.15.114:6790/0 conn(0x7fb4f40b4820 legacy=0x7fb4f413cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 >> v1:172.21.15.114:6789/0 conn(0x7fb4f40b3cd0 legacy=0x7fb4f413c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 monclient: found mon.b 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f41df550 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient: _finish_auth 0 2024-03-29T10:47:44.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.588+0000 7fb4ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4f3fff640 5 monclient: authenticate success, global_id 4504 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4f3fff640 10 monclient: _renew_subs 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4f3fff640 1 -- 172.21.15.114:0/1640344457 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f4071e20 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4f3fff640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4f3fff640 10 monclient: _renew_subs 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4f3fff640 1 -- 172.21.15.114:0/1640344457 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f40b0eb0 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4f3fff640 1 librados: init done 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4ec002010 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4f17fa640 4 set_mon_vals no callback set 2024-03-29T10:47:44.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb4ec004fb0 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.589+0000 7fb4ebfff640 10 monclient: dump: 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.591+0000 7fb4e9ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:44.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.591+0000 7fb4e9ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:44.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.591+0000 7fb4e9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:47:44.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.591+0000 7fb4e9ffb640 1 -- 172.21.15.114:0/1640344457 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc005080 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.591+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fb4ec01af80 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.592+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fb4ec084d10 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.592+0000 7fb4ebfff640 10 client.4504.objecter ms_dispatch 0x7fb4f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:44.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.592+0000 7fb4ebfff640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:44.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.592+0000 7fb4ebfff640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:44.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.592+0000 7fb4ebfff640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-29T10:47:44.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.592+0000 7fb4f27fc640 10 monclient: get_auth_request con 0x7fb4d406e160 auth_method 0 2024-03-29T10:47:44.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.592+0000 7fb4ebfff640 10 client.4504.objecter ms_handle_connect 0x7fb4d406e160 2024-03-29T10:47:44.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.601+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4ec050650 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.601+0000 7fb4ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:44.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.601+0000 7fb4ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:44.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.977+0000 7fb4e9ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:44.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.977+0000 7fb4e9ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:44.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.977+0000 7fb4e9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:47:44.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.977+0000 7fb4e9ffb640 1 -- 172.21.15.114:0/1640344457 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4cc005330 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.961 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.979+0000 7fb4ebfff640 1 -- 172.21.15.114:0/1640344457 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fb4ec0843c0 con 0x7fb4f40a9ce0 2024-03-29T10:47:44.961 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.979+0000 7fb4ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:44.961 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.979+0000 7fb4ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:44.962 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:47:44.962 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:44.964 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.982+0000 7fb4e9ffb640 10 librados: watch_flush enter 2024-03-29T10:47:44.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.983+0000 7fb4e9ffb640 10 librados: watch_flush exit 2024-03-29T10:47:44.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.983+0000 7fb4e9ffb640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-29T10:47:44.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.983+0000 7fb4e9ffb640 10 client.4504.objecter successfully canceled tick 2024-03-29T10:47:44.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.983+0000 7fb4e9ffb640 1 -- 172.21.15.114:0/1640344457 >> v1:172.21.15.114:6816/775044307 conn(0x7fb4d406e160 legacy=0x7fb4d4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.983+0000 7fb4e9ffb640 10 monclient: shutdown 2024-03-29T10:47:44.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.983+0000 7fb4e9ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:44.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.983+0000 7fb4e9ffb640 1 -- 172.21.15.114:0/1640344457 >> v1:172.21.15.176:6789/0 conn(0x7fb4f40a9ce0 legacy=0x7fb4f40ab410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.983+0000 7fb4e9ffb640 1 -- 172.21.15.114:0/1640344457 shutdown_connections 2024-03-29T10:47:44.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.983+0000 7fb4e9ffb640 1 -- 172.21.15.114:0/1640344457 >> 172.21.15.114:0/1640344457 conn(0x7fb4f406d070 msgr2=0x7fb4f4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:44.967 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.984+0000 7fb4e9ffb640 1 -- 172.21.15.114:0/1640344457 shutdown_connections 2024-03-29T10:47:44.967 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.984+0000 7fb4e9ffb640 1 -- 172.21.15.114:0/1640344457 wait complete. 2024-03-29T10:47:44.967 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:44.984+0000 7fb4e9ffb640 1 librados: shutdown 2024-03-29T10:47:44.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:51.788 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:51.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.951+0000 7f4daa18f640 10 monclient: get_monmap_and_config 2024-03-29T10:47:51.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.951+0000 7f4daa18f640 10 monclient: build_initial_monmap 2024-03-29T10:47:51.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.951+0000 7f4daa18f640 10 monclient: monmap: 2024-03-29T10:47:51.934 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:51.934 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:51.934 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:47:51.952604+0000 2024-03-29T10:47:51.934 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:47:51.952604+0000 2024-03-29T10:47:51.934 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:51.934 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:51.934 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:51.934 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:47:51.934 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:47:51.935 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:51.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.952+0000 7f4daa18f640 1 Processor -- start 2024-03-29T10:47:51.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.952+0000 7f4daa18f640 1 -- start start 2024-03-29T10:47:51.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.952+0000 7f4daa18f640 10 monclient: init 2024-03-29T10:47:51.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.953+0000 7f4daa18f640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:51.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.953+0000 7f4daa18f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:51.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.953+0000 7f4daa18f640 10 monclient(hunting): picked mon.noname-c con 0x7f4da41457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:51.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.953+0000 7f4daa18f640 10 monclient(hunting): picked mon.noname-b con 0x7f4da413f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:51.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.953+0000 7f4daa18f640 10 monclient(hunting): picked mon.noname-a con 0x7f4da40a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:51.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.953+0000 7f4daa18f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da40aaef0 con 0x7f4da40a9d70 2024-03-29T10:47:51.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.953+0000 7f4daa18f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da4145bd0 con 0x7f4da41457f0 2024-03-29T10:47:51.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.953+0000 7f4daa18f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da4145eb0 con 0x7f4da413f650 2024-03-29T10:47:51.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.953+0000 7f4daa18f640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:51.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.953+0000 7f4daa18f640 10 monclient(hunting): authenticate will time out at 1337.194779s 2024-03-29T10:47:51.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.958+0000 7f4da2ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f4da413f650 0x7f4da40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:37131/0 (socket says 172.21.15.114:37131) 2024-03-29T10:47:51.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.958+0000 7f4da2ffd640 1 -- 172.21.15.114:0/2504245700 learned_addr learned my addr 172.21.15.114:0/2504245700 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:51.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.959+0000 7f4da27fc640 1 -- 172.21.15.114:0/2504245700 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540896362 0 0) 0x7f4da4145eb0 con 0x7f4da413f650 2024-03-29T10:47:51.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.959+0000 7f4da27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.942 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.959+0000 7f4da27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.942 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.959+0000 7f4da27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.942 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.959+0000 7f4da27fc640 10 monclient(hunting): my global_id is 4610 2024-03-29T10:47:51.942 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.959+0000 7f4da27fc640 1 -- 172.21.15.114:0/2504245700 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8c003520 con 0x7f4da413f650 2024-03-29T10:47:51.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.963+0000 7f4da27fc640 1 -- 172.21.15.114:0/2504245700 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3983326507 0 0) 0x7f4d8c003520 con 0x7f4da413f650 2024-03-29T10:47:51.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.963+0000 7f4da27fc640 1 -- 172.21.15.114:0/2504245700 >> v1:172.21.15.176:6789/0 conn(0x7f4da41457f0 legacy=0x7f4da413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:51.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.963+0000 7f4da27fc640 1 -- 172.21.15.114:0/2504245700 >> v1:172.21.15.114:6789/0 conn(0x7f4da40a9d70 legacy=0x7f4da414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 1 monclient: found mon.noname-b 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 1 -- 172.21.15.114:0/2504245700 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da4146340 con 0x7f4da413f650 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 1 -- 172.21.15.114:0/2504245700 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4d9c001a10 con 0x7f4da413f650 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.964+0000 7f4da27fc640 10 monclient: dump: 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:51.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.965+0000 7f4da27fc640 1 -- 172.21.15.114:0/2504245700 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d9c001be0 con 0x7f4da413f650 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.965+0000 7f4daa18f640 5 monclient: authenticate success, global_id 4610 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.965+0000 7f4daa18f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.965+0000 7f4da27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.965+0000 7f4da27fc640 1 -- 172.21.15.114:0/2504245700 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4d9c002030 con 0x7f4da413f650 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.965+0000 7f4da27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.965+0000 7f4da27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.965+0000 7f4da27fc640 10 monclient: dump: 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:51.952 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:51.953 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:51.953 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:51.953 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:51.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.965+0000 7f4daa18f640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:51.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.965+0000 7f4daa18f640 4 set_mon_vals no callback set 2024-03-29T10:47:51.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.966+0000 7f4daa18f640 10 monclient: shutdown 2024-03-29T10:47:51.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.966+0000 7f4daa18f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:51.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.966+0000 7f4daa18f640 1 -- 172.21.15.114:0/2504245700 >> v1:172.21.15.114:6790/0 conn(0x7f4da413f650 legacy=0x7f4da40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.967+0000 7f4daa18f640 1 -- 172.21.15.114:0/2504245700 shutdown_connections 2024-03-29T10:47:51.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.967+0000 7f4daa18f640 1 -- 172.21.15.114:0/2504245700 >> 172.21.15.114:0/2504245700 conn(0x7f4da4066a00 msgr2=0x7f4da4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:51.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.967+0000 7f4daa18f640 1 -- 172.21.15.114:0/2504245700 shutdown_connections 2024-03-29T10:47:51.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.967+0000 7f4daa18f640 1 -- 172.21.15.114:0/2504245700 wait complete. 2024-03-29T10:47:51.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.968+0000 7f4daa18f640 10 monclient: build_initial_monmap 2024-03-29T10:47:51.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.968+0000 7f4daa18f640 10 monclient: monmap: 2024-03-29T10:47:51.954 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:51.954 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:51.954 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:47:51.954 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.968+0000 7f4daa18f640 1 librados: starting msgr at 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.968+0000 7f4daa18f640 1 librados: starting objecter 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.968+0000 7f4daa18f640 1 Processor -- start 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 1 -- start start 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 1 librados: setting wanted keys 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 1 librados: calling monclient init 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 10 monclient: init 2024-03-29T10:47:51.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:51.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:51.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 10 monclient(hunting): picked mon.noname-b con 0x7f4da40a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:51.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 10 monclient(hunting): picked mon.noname-a con 0x7f4da413f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:51.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 10 monclient(hunting): picked mon.noname-c con 0x7f4da41457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:51.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da413b370 con 0x7f4da413f650 2024-03-29T10:47:51.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da4139de0 con 0x7f4da40a9d70 2024-03-29T10:47:51.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da4139f70 con 0x7f4da41457f0 2024-03-29T10:47:51.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:51.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.969+0000 7f4daa18f640 10 monclient(hunting): authenticate will time out at 1337.210954s 2024-03-29T10:47:51.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.973+0000 7f4da3fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f4da41457f0 0x7f4da41e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:37197/0 (socket says 172.21.15.114:37197) 2024-03-29T10:47:51.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.973+0000 7f4da3fff640 1 -- 172.21.15.114:0/1486848913 learned_addr learned my addr 172.21.15.114:0/1486848913 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:51.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.973+0000 7f4d8bfff640 10 client.?.objecter ms_handle_connect 0x7f4da41457f0 2024-03-29T10:47:51.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.973+0000 7f4d8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.973+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695652602 0 0) 0x7f4da4139f70 con 0x7f4da41457f0 2024-03-29T10:47:51.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.973+0000 7f4d8bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.973+0000 7f4d8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.973+0000 7f4d8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.973+0000 7f4d8bfff640 10 monclient(hunting): my global_id is 4613 2024-03-29T10:47:51.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.973+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d90004eb0 con 0x7f4da41457f0 2024-03-29T10:47:51.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.974+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571908227 0 0) 0x7f4d90004eb0 con 0x7f4da41457f0 2024-03-29T10:47:51.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.974+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4da4139f70 con 0x7f4da41457f0 2024-03-29T10:47:51.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.974+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4d940017d0 con 0x7f4da41457f0 2024-03-29T10:47:51.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.974+0000 7f4d8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.974+0000 7f4d8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:51.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.974+0000 7f4d8bfff640 10 monclient(hunting): dump: 2024-03-29T10:47:51.959 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.974+0000 7f4d8bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.975+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3055242854 0 0) 0x7f4da4139f70 con 0x7f4da41457f0 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.975+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 >> v1:172.21.15.176:6789/0 conn(0x7f4da40a9d70 legacy=0x7f4da41396d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:51.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.975+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 >> v1:172.21.15.114:6789/0 conn(0x7f4da413f650 legacy=0x7f4da41dab20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:51.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.979+0000 7f4d8bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:51.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.979+0000 7f4d8bfff640 1 monclient: found mon.c 2024-03-29T10:47:51.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.979+0000 7f4d8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:51.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.979+0000 7f4d8bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:51.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.979+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da413b790 con 0x7f4da41457f0 2024-03-29T10:47:51.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.980+0000 7f4d8bfff640 10 monclient: _finish_auth 0 2024-03-29T10:47:51.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.980+0000 7f4d8bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:51.963 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.980+0000 7f4d8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:51.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.980+0000 7f4daa18f640 5 monclient: authenticate success, global_id 4613 2024-03-29T10:47:51.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.983+0000 7f4daa18f640 10 monclient: _renew_subs 2024-03-29T10:47:51.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.983+0000 7f4daa18f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:51.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.983+0000 7f4daa18f640 1 -- 172.21.15.114:0/1486848913 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da413b9e0 con 0x7f4da41457f0 2024-03-29T10:47:51.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.983+0000 7f4daa18f640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:51.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.983+0000 7f4daa18f640 10 monclient: _renew_subs 2024-03-29T10:47:51.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.983+0000 7f4daa18f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.983+0000 7f4daa18f640 1 -- 172.21.15.114:0/1486848913 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da41db410 con 0x7f4da41457f0 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.983+0000 7f4daa18f640 1 librados: init done 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.985+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d94001d80 con 0x7f4da41457f0 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.985+0000 7f4d8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.985+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4d94004a40 con 0x7f4da41457f0 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.985+0000 7f4d8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.985+0000 7f4d8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.985+0000 7f4d8bfff640 10 monclient: dump: 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.969 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:51.970 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:51.970 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:51.970 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:51.970 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:51.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.985+0000 7f4d89ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.985+0000 7f4d89ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.985+0000 7f4d89ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:51.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.985+0000 7f4d89ffb640 1 -- 172.21.15.114:0/1486848913 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da41db710 con 0x7f4da41457f0 2024-03-29T10:47:51.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.987+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f4d94004ba0 con 0x7f4da41457f0 2024-03-29T10:47:51.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.987+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f4d94084820 con 0x7f4da41457f0 2024-03-29T10:47:51.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.987+0000 7f4d8bfff640 10 client.4613.objecter ms_dispatch 0x7f4da4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:51.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.987+0000 7f4d8bfff640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:51.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.987+0000 7f4d8bfff640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:51.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.987+0000 7f4da1ffb640 4 set_mon_vals no callback set 2024-03-29T10:47:51.971 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.988+0000 7f4da37fe640 10 monclient: get_auth_request con 0x7f4d9006dc40 auth_method 0 2024-03-29T10:47:51.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.988+0000 7f4d8bfff640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T10:47:51.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.988+0000 7f4d8bfff640 10 client.4613.objecter ms_handle_connect 0x7f4d9006dc40 2024-03-29T10:47:51.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.994+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d9404ff30 con 0x7f4da41457f0 2024-03-29T10:47:51.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.994+0000 7f4d8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:51.994+0000 7f4d8bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:52.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.377+0000 7f4d89ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:52.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.377+0000 7f4d89ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:52.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.377+0000 7f4d89ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:47:52.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.377+0000 7f4d89ffb640 1 -- 172.21.15.114:0/1486848913 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4da41db710 con 0x7f4da41457f0 2024-03-29T10:47:52.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.378+0000 7f4d8bfff640 1 -- 172.21.15.114:0/1486848913 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f4da41db710 con 0x7f4da41457f0 2024-03-29T10:47:52.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.378+0000 7f4d8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:52.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.378+0000 7f4d8bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:52.361 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:47:52.361 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:52.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.382+0000 7f4d89ffb640 10 librados: watch_flush enter 2024-03-29T10:47:52.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.382+0000 7f4d89ffb640 10 librados: watch_flush exit 2024-03-29T10:47:52.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.382+0000 7f4d89ffb640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T10:47:52.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.382+0000 7f4d89ffb640 10 client.4613.objecter successfully canceled tick 2024-03-29T10:47:52.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.382+0000 7f4d89ffb640 1 -- 172.21.15.114:0/1486848913 >> v1:172.21.15.114:6816/775044307 conn(0x7f4d9006dc40 legacy=0x7f4d90070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.382+0000 7f4d89ffb640 10 monclient: shutdown 2024-03-29T10:47:52.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.382+0000 7f4d89ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:52.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.382+0000 7f4d89ffb640 1 -- 172.21.15.114:0/1486848913 >> v1:172.21.15.114:6790/0 conn(0x7f4da41457f0 legacy=0x7f4da41e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.382+0000 7f4d89ffb640 1 -- 172.21.15.114:0/1486848913 shutdown_connections 2024-03-29T10:47:52.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.382+0000 7f4d89ffb640 1 -- 172.21.15.114:0/1486848913 >> 172.21.15.114:0/1486848913 conn(0x7f4da4066a00 msgr2=0x7f4da40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:52.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.383+0000 7f4d89ffb640 1 -- 172.21.15.114:0/1486848913 shutdown_connections 2024-03-29T10:47:52.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.383+0000 7f4d89ffb640 1 -- 172.21.15.114:0/1486848913 wait complete. 2024-03-29T10:47:52.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:52.383+0000 7f4d89ffb640 1 librados: shutdown 2024-03-29T10:47:52.380 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:59.187 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:59.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.357+0000 7ff6f2a9a640 10 monclient: get_monmap_and_config 2024-03-29T10:47:59.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.357+0000 7ff6f2a9a640 10 monclient: build_initial_monmap 2024-03-29T10:47:59.343 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.357+0000 7ff6f2a9a640 10 monclient: monmap: 2024-03-29T10:47:59.343 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:59.343 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:59.343 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:47:59.358689+0000 2024-03-29T10:47:59.343 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:47:59.358689+0000 2024-03-29T10:47:59.343 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:59.343 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:59.343 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:59.343 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:47:59.343 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:47:59.343 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:59.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.359+0000 7ff6f2a9a640 1 Processor -- start 2024-03-29T10:47:59.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.359+0000 7ff6f2a9a640 1 -- start start 2024-03-29T10:47:59.344 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.359+0000 7ff6f2a9a640 10 monclient: init 2024-03-29T10:47:59.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.360+0000 7ff6f2a9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:59.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.360+0000 7ff6f2a9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:59.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.360+0000 7ff6f2a9a640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ec0ab360 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:59.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.360+0000 7ff6f2a9a640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ec14fb00 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:59.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.364+0000 7ff6f0836640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ff6ec0ab360 0x7ff6ec0ad450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:37227/0 (socket says 172.21.15.114:37227) 2024-03-29T10:47:59.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.364+0000 7ff6f2a9a640 10 monclient(hunting): picked mon.noname-a con 0x7ff6ec153090 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:59.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.364+0000 7ff6f0836640 1 -- 172.21.15.114:0/1078267039 learned_addr learned my addr 172.21.15.114:0/1078267039 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.364+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/1078267039 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec0a94e0 con 0x7ff6ec153090 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.364+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/1078267039 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec0afb60 con 0x7ff6ec0ab360 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.364+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/1078267039 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec0aab80 con 0x7ff6ec14fb00 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.364+0000 7ff6f2a9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.364+0000 7ff6f2a9a640 10 monclient(hunting): authenticate will time out at 1344.605528s 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 1 -- 172.21.15.114:0/1078267039 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694308326 0 0) 0x7ff6ec0a94e0 con 0x7ff6ec153090 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 10 monclient(hunting): my global_id is 4533 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 1 -- 172.21.15.114:0/1078267039 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d4003520 con 0x7ff6ec153090 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 1 -- 172.21.15.114:0/1078267039 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 170464257 0 0) 0x7ff6d4003520 con 0x7ff6ec153090 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 1 -- 172.21.15.114:0/1078267039 >> v1:172.21.15.114:6790/0 conn(0x7ff6ec14fb00 legacy=0x7ff6ec152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 1 -- 172.21.15.114:0/1078267039 >> v1:172.21.15.176:6789/0 conn(0x7ff6ec0ab360 legacy=0x7ff6ec0ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 1 monclient: found mon.noname-a 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 1 -- 172.21.15.114:0/1078267039 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec0aa680 con 0x7ff6ec153090 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 1 -- 172.21.15.114:0/1078267039 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff6e4001a10 con 0x7ff6ec153090 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6eb7fe640 10 monclient: dump: 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6f2a9a640 5 monclient: authenticate success, global_id 4533 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.367+0000 7ff6f2a9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.369+0000 7ff6eb7fe640 1 -- 172.21.15.114:0/1078267039 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6e4001ca0 con 0x7ff6ec153090 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.369+0000 7ff6eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.369+0000 7ff6eb7fe640 1 -- 172.21.15.114:0/1078267039 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff6e4002030 con 0x7ff6ec153090 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.369+0000 7ff6eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.369+0000 7ff6eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.369+0000 7ff6eb7fe640 10 monclient: dump: 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:59.355 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:59.355 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:59.355 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:59.355 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:59.355 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:59.355 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:59.355 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:59.355 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:59.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.371+0000 7ff6f2a9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:59.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.371+0000 7ff6f2a9a640 4 set_mon_vals no callback set 2024-03-29T10:47:59.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.372+0000 7ff6f2a9a640 10 monclient: shutdown 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.372+0000 7ff6f2a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.372+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/1078267039 >> v1:172.21.15.114:6789/0 conn(0x7ff6ec153090 legacy=0x7ff6ec15cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.372+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/1078267039 shutdown_connections 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.372+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/1078267039 >> 172.21.15.114:0/1078267039 conn(0x7ff6ec0b35e0 msgr2=0x7ff6ec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.372+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/1078267039 shutdown_connections 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.372+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/1078267039 wait complete. 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.373+0000 7ff6f2a9a640 10 monclient: build_initial_monmap 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.373+0000 7ff6f2a9a640 10 monclient: monmap: 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:59.357 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:47:59.358 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:47:59.358 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:47:59.358 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:59.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.374+0000 7ff6f2a9a640 1 librados: starting msgr at 2024-03-29T10:47:59.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.374+0000 7ff6f2a9a640 1 librados: starting objecter 2024-03-29T10:47:59.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.374+0000 7ff6f2a9a640 1 Processor -- start 2024-03-29T10:47:59.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.374+0000 7ff6f2a9a640 1 -- start start 2024-03-29T10:47:59.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.374+0000 7ff6f2a9a640 1 librados: setting wanted keys 2024-03-29T10:47:59.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.374+0000 7ff6f2a9a640 1 librados: calling monclient init 2024-03-29T10:47:59.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.374+0000 7ff6f2a9a640 10 monclient: init 2024-03-29T10:47:59.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.376+0000 7ff6f2a9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:59.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.376+0000 7ff6f2a9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:59.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.376+0000 7ff6f2a9a640 10 monclient(hunting): picked mon.noname-a con 0x7ff6ec0ab360 addr v1:172.21.15.114:6789/0 2024-03-29T10:47:59.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.379+0000 7ff6f2a9a640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ec14fb00 addr v1:172.21.15.114:6790/0 2024-03-29T10:47:59.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.379+0000 7ff6ebfff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff6ec14fb00 0x7ff6ec06eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:58021/0 (socket says 172.21.15.114:58021) 2024-03-29T10:47:59.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.379+0000 7ff6f0836640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff6ec0ab360 0x7ff6ec06fb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58975/0 (socket says 172.21.15.114:58975) 2024-03-29T10:47:59.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.379+0000 7ff6ebfff640 1 -- 172.21.15.114:0/2371296358 learned_addr learned my addr 172.21.15.114:0/2371296358 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:47:59.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.379+0000 7ff6f2a9a640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ec153090 addr v1:172.21.15.176:6789/0 2024-03-29T10:47:59.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.379+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec0712a0 con 0x7ff6ec0ab360 2024-03-29T10:47:59.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.379+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec0873c0 con 0x7ff6ec153090 2024-03-29T10:47:59.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.380+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec08ae30 con 0x7ff6ec14fb00 2024-03-29T10:47:59.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.380+0000 7ff6f2a9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:59.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.380+0000 7ff6f2a9a640 10 monclient(hunting): authenticate will time out at 1344.621176s 2024-03-29T10:47:59.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.380+0000 7ff6e97fa640 10 client.?.objecter ms_handle_connect 0x7ff6ec14fb00 2024-03-29T10:47:59.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.380+0000 7ff6e97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:59.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.380+0000 7ff6e97fa640 10 client.?.objecter ms_handle_connect 0x7ff6ec0ab360 2024-03-29T10:47:59.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.380+0000 7ff6e97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:59.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066843909 0 0) 0x7ff6ec0712a0 con 0x7ff6ec0ab360 2024-03-29T10:47:59.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:59.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:59.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:59.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 10 monclient(hunting): my global_id is 4536 2024-03-29T10:47:59.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d8004eb0 con 0x7ff6ec0ab360 2024-03-29T10:47:59.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4197755 0 0) 0x7ff6d8004eb0 con 0x7ff6ec0ab360 2024-03-29T10:47:59.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6ec0712a0 con 0x7ff6ec0ab360 2024-03-29T10:47:59.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff6dc001510 con 0x7ff6ec0ab360 2024-03-29T10:47:59.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:59.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 10 monclient(hunting): dump: 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.382+0000 7ff6e97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.383+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1524899884 0 0) 0x7ff6ec0712a0 con 0x7ff6ec0ab360 2024-03-29T10:47:59.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.383+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 >> v1:172.21.15.114:6790/0 conn(0x7ff6ec14fb00 legacy=0x7ff6ec06eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.383+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 >> v1:172.21.15.176:6789/0 conn(0x7ff6ec153090 legacy=0x7ff6ec08a720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:59.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6e97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6e97fa640 1 monclient: found mon.a 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6e97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec08afc0 con 0x7ff6ec0ab360 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6e97fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6e97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6f2a9a640 5 monclient: authenticate success, global_id 4536 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6f2a9a640 10 monclient: _renew_subs 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6f2a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ec0878c0 con 0x7ff6ec0ab360 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6f2a9a640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6f2a9a640 10 monclient: _renew_subs 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6f2a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ec087e40 con 0x7ff6ec0ab360 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.387+0000 7ff6f2a9a640 1 librados: init done 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.388+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6dc001a80 con 0x7ff6ec0ab360 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.388+0000 7ff6e97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.388+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7ff6dc004a90 con 0x7ff6ec0ab360 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.388+0000 7ff6e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.388+0000 7ff6e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.388+0000 7ff6e97fa640 10 monclient: dump: 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.388+0000 7ff6f2a9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.388+0000 7ff6ea7fc640 4 set_mon_vals no callback set 2024-03-29T10:47:59.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.389+0000 7ff6f2a9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:59.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.389+0000 7ff6f2a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:59.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.389+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ec087550 con 0x7ff6ec0ab360 2024-03-29T10:47:59.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.390+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7ff6dc01aa60 con 0x7ff6ec0ab360 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.390+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7ff6dc084a30 con 0x7ff6ec0ab360 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.390+0000 7ff6e97fa640 10 client.4536.objecter ms_dispatch 0x7ff6ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.390+0000 7ff6e97fa640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.390+0000 7ff6e97fa640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.390+0000 7ff6e97fa640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.391+0000 7ff6ebfff640 10 monclient: get_auth_request con 0x7ff6d806dc80 auth_method 0 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.391+0000 7ff6e97fa640 10 client.4536.objecter ms_handle_connect 0x7ff6d806dc80 2024-03-29T10:47:59.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.400+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6dc050310 con 0x7ff6ec0ab360 2024-03-29T10:47:59.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.400+0000 7ff6e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:59.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.400+0000 7ff6e97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:59.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.720+0000 7ff6f2a9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:59.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.720+0000 7ff6f2a9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:59.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.720+0000 7ff6f2a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:47:59.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.720+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6ec08b150 con 0x7ff6ec0ab360 2024-03-29T10:47:59.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.722+0000 7ff6e97fa640 1 -- 172.21.15.114:0/2371296358 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7ff6dc052240 con 0x7ff6ec0ab360 2024-03-29T10:47:59.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.722+0000 7ff6e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:59.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.722+0000 7ff6e97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:59.708 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:47:59.708 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:59.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.726+0000 7ff6f2a9a640 10 librados: watch_flush enter 2024-03-29T10:47:59.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.726+0000 7ff6f2a9a640 10 librados: watch_flush exit 2024-03-29T10:47:59.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.726+0000 7ff6f2a9a640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-29T10:47:59.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.726+0000 7ff6f2a9a640 10 client.4536.objecter successfully canceled tick 2024-03-29T10:47:59.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.727+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 >> v1:172.21.15.114:6816/775044307 conn(0x7ff6d806dc80 legacy=0x7ff6d8070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.727+0000 7ff6f2a9a640 10 monclient: shutdown 2024-03-29T10:47:59.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.727+0000 7ff6f2a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:59.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.727+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 >> v1:172.21.15.114:6789/0 conn(0x7ff6ec0ab360 legacy=0x7ff6ec06fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.727+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 shutdown_connections 2024-03-29T10:47:59.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.727+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 >> 172.21.15.114:0/2371296358 conn(0x7ff6ec0b35e0 msgr2=0x7ff6ec14ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:59.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.728+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 shutdown_connections 2024-03-29T10:47:59.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.728+0000 7ff6f2a9a640 1 -- 172.21.15.114:0/2371296358 wait complete. 2024-03-29T10:47:59.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:47:59.729+0000 7ff6f2a9a640 1 librados: shutdown 2024-03-29T10:47:59.729 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:06.536 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:06.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.695+0000 7f20f6063640 10 monclient: get_monmap_and_config 2024-03-29T10:48:06.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.695+0000 7f20f6063640 10 monclient: build_initial_monmap 2024-03-29T10:48:06.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.695+0000 7f20f6063640 10 monclient: monmap: 2024-03-29T10:48:06.679 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:06.680 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:06.680 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:48:06.697129+0000 2024-03-29T10:48:06.680 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:48:06.697129+0000 2024-03-29T10:48:06.680 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:06.680 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:06.680 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:06.680 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:48:06.680 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:48:06.680 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:06.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.698+0000 7f20f6063640 1 Processor -- start 2024-03-29T10:48:06.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.698+0000 7f20f6063640 1 -- start start 2024-03-29T10:48:06.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.698+0000 7f20f6063640 10 monclient: init 2024-03-29T10:48:06.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.699+0000 7f20f6063640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:06.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.699+0000 7f20f6063640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:06.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.699+0000 7f20f6063640 10 monclient(hunting): picked mon.noname-a con 0x7f20f00accf0 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:06.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.702+0000 7f20f6063640 10 monclient(hunting): picked mon.noname-b con 0x7f20f00ad880 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:06.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.702+0000 7f20f6063640 10 monclient(hunting): picked mon.noname-c con 0x7f20f00ad0d0 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:06.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.702+0000 7f20eeffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f20f00ad880 0x7f20f013d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:52861/0 (socket says 172.21.15.114:52861) 2024-03-29T10:48:06.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.702+0000 7f20eeffd640 1 -- 172.21.15.114:0/1230185230 learned_addr learned my addr 172.21.15.114:0/1230185230 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:06.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.703+0000 7f20f6063640 1 -- 172.21.15.114:0/1230185230 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f00ad500 con 0x7f20f00accf0 2024-03-29T10:48:06.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.703+0000 7f20f6063640 1 -- 172.21.15.114:0/1230185230 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f00aec00 con 0x7f20f00ad0d0 2024-03-29T10:48:06.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.703+0000 7f20f6063640 1 -- 172.21.15.114:0/1230185230 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f00aeee0 con 0x7f20f00ad880 2024-03-29T10:48:06.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.703+0000 7f20f6063640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:06.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.703+0000 7f20f6063640 10 monclient(hunting): authenticate will time out at 1351.944518s 2024-03-29T10:48:06.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.703+0000 7f20ee7fc640 1 -- 172.21.15.114:0/1230185230 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828863143 0 0) 0x7f20f00aeee0 con 0x7f20f00ad880 2024-03-29T10:48:06.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.703+0000 7f20ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:06.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.703+0000 7f20ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:06.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.703+0000 7f20ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:06.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.703+0000 7f20ee7fc640 10 monclient(hunting): my global_id is 4622 2024-03-29T10:48:06.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.703+0000 7f20ee7fc640 1 -- 172.21.15.114:0/1230185230 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20d8003520 con 0x7f20f00ad880 2024-03-29T10:48:06.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 1 -- 172.21.15.114:0/1230185230 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102549313 0 0) 0x7f20f00aec00 con 0x7f20f00ad0d0 2024-03-29T10:48:06.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:06.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:06.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:06.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 10 monclient(hunting): my global_id is 4507 2024-03-29T10:48:06.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 1 -- 172.21.15.114:0/1230185230 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f00aeee0 con 0x7f20f00ad0d0 2024-03-29T10:48:06.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 1 -- 172.21.15.114:0/1230185230 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2903021057 0 0) 0x7f20d8003520 con 0x7f20f00ad880 2024-03-29T10:48:06.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 1 -- 172.21.15.114:0/1230185230 >> v1:172.21.15.176:6789/0 conn(0x7f20f00ad0d0 legacy=0x7f20f014b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 1 -- 172.21.15.114:0/1230185230 >> v1:172.21.15.114:6789/0 conn(0x7f20f00accf0 legacy=0x7f20f00af8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:06.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:06.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 1 monclient: found mon.noname-b 2024-03-29T10:48:06.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.704+0000 7f20ee7fc640 1 -- 172.21.15.114:0/1230185230 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f00af1c0 con 0x7f20f00ad880 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 1 -- 172.21.15.114:0/1230185230 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f20e4001690 con 0x7f20f00ad880 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 10 monclient: dump: 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:06.694 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20f6063640 5 monclient: authenticate success, global_id 4622 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20f6063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 1 -- 172.21.15.114:0/1230185230 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20e40018b0 con 0x7f20f00ad880 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 1 -- 172.21.15.114:0/1230185230 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f20e4001c40 con 0x7f20f00ad880 2024-03-29T10:48:06.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20f6063640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20f6063640 4 set_mon_vals no callback set 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.705+0000 7f20ee7fc640 10 monclient: dump: 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:06.696 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:06.697 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:06.697 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:06.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.706+0000 7f20f6063640 10 monclient: shutdown 2024-03-29T10:48:06.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.706+0000 7f20f6063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:06.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.706+0000 7f20f6063640 1 -- 172.21.15.114:0/1230185230 >> v1:172.21.15.114:6790/0 conn(0x7f20f00ad880 legacy=0x7f20f013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.706+0000 7f20f6063640 1 -- 172.21.15.114:0/1230185230 shutdown_connections 2024-03-29T10:48:06.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.706+0000 7f20f6063640 1 -- 172.21.15.114:0/1230185230 >> 172.21.15.114:0/1230185230 conn(0x7f20f00b35e0 msgr2=0x7f20f00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:06.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.708+0000 7f20f6063640 1 -- 172.21.15.114:0/1230185230 shutdown_connections 2024-03-29T10:48:06.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.708+0000 7f20f6063640 1 -- 172.21.15.114:0/1230185230 wait complete. 2024-03-29T10:48:06.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.709+0000 7f20f6063640 10 monclient: build_initial_monmap 2024-03-29T10:48:06.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.709+0000 7f20f6063640 10 monclient: monmap: 2024-03-29T10:48:06.701 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:06.701 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:06.701 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:48:06.701 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:48:06.702 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:06.702 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:06.702 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:06.702 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:48:06.702 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:48:06.702 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:06.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.709+0000 7f20f6063640 1 librados: starting msgr at 2024-03-29T10:48:06.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.709+0000 7f20f6063640 1 librados: starting objecter 2024-03-29T10:48:06.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.710+0000 7f20f6063640 1 Processor -- start 2024-03-29T10:48:06.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.710+0000 7f20f6063640 1 -- start start 2024-03-29T10:48:06.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.710+0000 7f20f6063640 1 librados: setting wanted keys 2024-03-29T10:48:06.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.710+0000 7f20f6063640 1 librados: calling monclient init 2024-03-29T10:48:06.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.710+0000 7f20f6063640 10 monclient: init 2024-03-29T10:48:06.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.711+0000 7f20f6063640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:06.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.711+0000 7f20f6063640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:06.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.711+0000 7f20f6063640 10 monclient(hunting): picked mon.noname-c con 0x7f20f00accf0 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:06.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.711+0000 7f20f6063640 10 monclient(hunting): picked mon.noname-a con 0x7f20f00ad0d0 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.711+0000 7f20f6063640 10 monclient(hunting): picked mon.noname-b con 0x7f20f00ad880 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.711+0000 7f20f6063640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f01e45d0 con 0x7f20f00ad0d0 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.711+0000 7f20f6063640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f013a680 con 0x7f20f00ad880 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.711+0000 7f20f6063640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f01e20a0 con 0x7f20f00accf0 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.711+0000 7f20f6063640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.711+0000 7f20f6063640 10 monclient(hunting): authenticate will time out at 1351.952200s 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20effff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f20f00ad880 0x7f20f0138df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:43224/0 (socket says 172.21.15.114:43224) 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20effff640 1 -- 172.21.15.114:0/3222489413 learned_addr learned my addr 172.21.15.114:0/3222489413 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 10 client.?.objecter ms_handle_connect 0x7f20f00ad880 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216626616 0 0) 0x7f20f013a680 con 0x7f20f00ad880 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:06.704 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 10 monclient(hunting): my global_id is 4510 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20dc004eb0 con 0x7f20f00ad880 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2865483742 0 0) 0x7f20dc004eb0 con 0x7f20f00ad880 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20f013a680 con 0x7f20f00ad880 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f20e0001950 con 0x7f20f00ad880 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.715+0000 7f20d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.716+0000 7f20d7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.716+0000 7f20d7fff640 10 monclient(hunting): dump: 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.705 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.716+0000 7f20d7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.716+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1015574340 0 0) 0x7f20f013a680 con 0x7f20f00ad880 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.716+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 >> v1:172.21.15.114:6790/0 conn(0x7f20f00accf0 legacy=0x7f20f013c500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.716+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 >> v1:172.21.15.114:6789/0 conn(0x7f20f00ad0d0 legacy=0x7f20f01e4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.716+0000 7f20d7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.716+0000 7f20d7fff640 1 monclient: found mon.b 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.716+0000 7f20d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.716+0000 7f20d7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f013cc10 con 0x7f20f00ad880 2024-03-29T10:48:06.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 10 monclient: _finish_auth 0 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20f6063640 5 monclient: authenticate success, global_id 4510 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20f6063640 10 monclient: _renew_subs 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20f6063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20f6063640 1 -- 172.21.15.114:0/3222489413 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f0071110 con 0x7f20f00ad880 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 10 client.4510.objecter ms_handle_connect 0x7f20f00ad0d0 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20f6063640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20f6063640 10 monclient: _renew_subs 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20f6063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20f6063640 1 -- 172.21.15.114:0/3222489413 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f006e980 con 0x7f20f00ad880 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20f6063640 1 librados: init done 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 10 client.4510.objecter resend_mon_ops 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20e0001d10 con 0x7f20f00ad880 2024-03-29T10:48:06.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f20e0003a60 con 0x7f20f00ad880 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20d7fff640 10 monclient: dump: 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:06.708 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:06.709 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:06.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.717+0000 7f20edffb640 4 set_mon_vals no callback set 2024-03-29T10:48:06.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.719+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f20e0003bc0 con 0x7f20f00ad880 2024-03-29T10:48:06.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.720+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f20e0084b20 con 0x7f20f00ad880 2024-03-29T10:48:06.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.720+0000 7f20d7fff640 10 client.4510.objecter ms_dispatch 0x7f20f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:06.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.720+0000 7f20d7fff640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:06.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.720+0000 7f20d7fff640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:06.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.720+0000 7f20d7fff640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-29T10:48:06.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.720+0000 7f20f6063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:06.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.720+0000 7f20f6063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:06.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.720+0000 7f20f6063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:06.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.720+0000 7f20f6063640 1 -- 172.21.15.114:0/3222489413 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d0005080 con 0x7f20f00ad880 2024-03-29T10:48:06.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.722+0000 7f20ef7fe640 10 monclient: get_auth_request con 0x7f20dc06dc60 auth_method 0 2024-03-29T10:48:06.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.724+0000 7f20d7fff640 10 client.4510.objecter ms_handle_connect 0x7f20dc06dc60 2024-03-29T10:48:06.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.729+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20e0050230 con 0x7f20f00ad880 2024-03-29T10:48:06.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.729+0000 7f20d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:06.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:06.729+0000 7f20d7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:07.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.040+0000 7f20f6063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:07.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.040+0000 7f20f6063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:07.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.040+0000 7f20f6063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:07.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.040+0000 7f20f6063640 1 -- 172.21.15.114:0/3222489413 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20d0005330 con 0x7f20f00ad880 2024-03-29T10:48:07.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.042+0000 7f20d7fff640 1 -- 172.21.15.114:0/3222489413 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f20e0052330 con 0x7f20f00ad880 2024-03-29T10:48:07.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.042+0000 7f20d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:07.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.042+0000 7f20d7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:07.025 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:48:07.026 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:07.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.046+0000 7f20f6063640 10 librados: watch_flush enter 2024-03-29T10:48:07.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.047+0000 7f20f6063640 10 librados: watch_flush exit 2024-03-29T10:48:07.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.047+0000 7f20f6063640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-29T10:48:07.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.047+0000 7f20f6063640 10 client.4510.objecter successfully canceled tick 2024-03-29T10:48:07.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.047+0000 7f20f6063640 1 -- 172.21.15.114:0/3222489413 >> v1:172.21.15.114:6816/775044307 conn(0x7f20dc06dc60 legacy=0x7f20dc070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:07.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.047+0000 7f20f6063640 10 monclient: shutdown 2024-03-29T10:48:07.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.047+0000 7f20f6063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:07.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.047+0000 7f20f6063640 1 -- 172.21.15.114:0/3222489413 >> v1:172.21.15.176:6789/0 conn(0x7f20f00ad880 legacy=0x7f20f0138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:07.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.049+0000 7f20f6063640 1 -- 172.21.15.114:0/3222489413 shutdown_connections 2024-03-29T10:48:07.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.049+0000 7f20f6063640 1 -- 172.21.15.114:0/3222489413 >> 172.21.15.114:0/3222489413 conn(0x7f20f00b35e0 msgr2=0x7f20f00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:07.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.050+0000 7f20f6063640 1 -- 172.21.15.114:0/3222489413 shutdown_connections 2024-03-29T10:48:07.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.050+0000 7f20f6063640 1 -- 172.21.15.114:0/3222489413 wait complete. 2024-03-29T10:48:07.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:07.050+0000 7f20f6063640 1 librados: shutdown 2024-03-29T10:48:07.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:13.859 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:14.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.024+0000 7f2a4a932640 10 monclient: get_monmap_and_config 2024-03-29T10:48:14.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.024+0000 7f2a4a932640 10 monclient: build_initial_monmap 2024-03-29T10:48:14.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.024+0000 7f2a4a932640 10 monclient: monmap: 2024-03-29T10:48:14.010 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:14.010 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:14.010 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:48:14.025945+0000 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:48:14.025945+0000 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.026+0000 7f2a4a932640 1 Processor -- start 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.026+0000 7f2a4a932640 1 -- start start 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.026+0000 7f2a4a932640 10 monclient: init 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.027+0000 7f2a4a932640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.027+0000 7f2a4a932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.027+0000 7f2a4a932640 10 monclient(hunting): picked mon.noname-a con 0x7f2a44070170 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.027+0000 7f2a4a932640 10 monclient(hunting): picked mon.noname-c con 0x7f2a440713f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.027+0000 7f2a4a932640 10 monclient(hunting): picked mon.noname-b con 0x7f2a44070550 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.027+0000 7f2a4a932640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4413dbd0 con 0x7f2a44070170 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.027+0000 7f2a4a932640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4413ff70 con 0x7f2a440713f0 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.027+0000 7f2a4a932640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a44140250 con 0x7f2a44070550 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.027+0000 7f2a4a932640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.027+0000 7f2a4a932640 10 monclient(hunting): authenticate will time out at 1359.268449s 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.028+0000 7f2a48ecf640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f2a44070550 0x7f2a4414b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:41318/0 (socket says 172.21.15.114:41318) 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.028+0000 7f2a48ecf640 1 -- 172.21.15.114:0/4242450262 learned_addr learned my addr 172.21.15.114:0/4242450262 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:14.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 1 -- 172.21.15.114:0/4242450262 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893243029 0 0) 0x7f2a44140250 con 0x7f2a44070550 2024-03-29T10:48:14.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:14.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:14.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:14.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 10 monclient(hunting): my global_id is 4625 2024-03-29T10:48:14.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 1 -- 172.21.15.114:0/4242450262 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a30003520 con 0x7f2a44070550 2024-03-29T10:48:14.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 1 -- 172.21.15.114:0/4242450262 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2516439326 0 0) 0x7f2a30003520 con 0x7f2a44070550 2024-03-29T10:48:14.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 1 -- 172.21.15.114:0/4242450262 >> v1:172.21.15.176:6789/0 conn(0x7f2a440713f0 legacy=0x7f2a441406e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 1 -- 172.21.15.114:0/4242450262 >> v1:172.21.15.114:6789/0 conn(0x7f2a44070170 legacy=0x7f2a4413d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 1 monclient: found mon.noname-b 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 1 -- 172.21.15.114:0/4242450262 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a44140df0 con 0x7f2a44070550 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 1 -- 172.21.15.114:0/4242450262 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2a2c001690 con 0x7f2a44070550 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.032+0000 7f2a42ffd640 10 monclient: dump: 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.033+0000 7f2a42ffd640 1 -- 172.21.15.114:0/4242450262 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a2c001860 con 0x7f2a44070550 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.033+0000 7f2a42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.033+0000 7f2a42ffd640 1 -- 172.21.15.114:0/4242450262 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2a2c001cb0 con 0x7f2a44070550 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.033+0000 7f2a42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.033+0000 7f2a42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.033+0000 7f2a42ffd640 10 monclient: dump: 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.033+0000 7f2a4a932640 5 monclient: authenticate success, global_id 4625 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.033+0000 7f2a4a932640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.033+0000 7f2a4a932640 4 set_mon_vals no callback set 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.034+0000 7f2a4a932640 10 monclient: shutdown 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.034+0000 7f2a4a932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.034+0000 7f2a4a932640 1 -- 172.21.15.114:0/4242450262 >> v1:172.21.15.114:6790/0 conn(0x7f2a44070550 legacy=0x7f2a4414b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.034+0000 7f2a4a932640 1 -- 172.21.15.114:0/4242450262 shutdown_connections 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.034+0000 7f2a4a932640 1 -- 172.21.15.114:0/4242450262 >> 172.21.15.114:0/4242450262 conn(0x7f2a44067610 msgr2=0x7f2a440b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.034+0000 7f2a4a932640 1 -- 172.21.15.114:0/4242450262 shutdown_connections 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.034+0000 7f2a4a932640 1 -- 172.21.15.114:0/4242450262 wait complete. 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.035+0000 7f2a4a932640 10 monclient: build_initial_monmap 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.035+0000 7f2a4a932640 10 monclient: monmap: 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.035+0000 7f2a4a932640 1 librados: starting msgr at 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.035+0000 7f2a4a932640 1 librados: starting objecter 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.035+0000 7f2a4a932640 1 Processor -- start 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.036+0000 7f2a4a932640 1 -- start start 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.036+0000 7f2a4a932640 1 librados: setting wanted keys 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.036+0000 7f2a4a932640 1 librados: calling monclient init 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.036+0000 7f2a4a932640 10 monclient: init 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.037+0000 7f2a4a932640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.037+0000 7f2a4a932640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.037+0000 7f2a4a932640 10 monclient(hunting): picked mon.noname-a con 0x7f2a44070170 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.037+0000 7f2a4a932640 10 monclient(hunting): picked mon.noname-c con 0x7f2a44070550 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.038+0000 7f2a4a932640 10 monclient(hunting): picked mon.noname-b con 0x7f2a440713f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.038+0000 7f2a4a932640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4413a830 con 0x7f2a44070170 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.038+0000 7f2a4a932640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4413acc0 con 0x7f2a440713f0 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.038+0000 7f2a4a932640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a441d5cc0 con 0x7f2a44070550 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.038+0000 7f2a4a932640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.038+0000 7f2a4a932640 10 monclient(hunting): authenticate will time out at 1359.279257s 2024-03-29T10:48:14.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.040+0000 7f2a43fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2a44070170 0x7f2a441d1f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39109/0 (socket says 172.21.15.114:39109) 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.040+0000 7f2a43fff640 1 -- 172.21.15.114:0/3824310386 learned_addr learned my addr 172.21.15.114:0/3824310386 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 client.?.objecter ms_handle_connect 0x7f2a44070170 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 client.?.objecter ms_handle_connect 0x7f2a440713f0 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 client.?.objecter ms_handle_connect 0x7f2a44070550 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752403658 0 0) 0x7f2a4413acc0 con 0x7f2a440713f0 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): my global_id is 4513 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a24004eb0 con 0x7f2a440713f0 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534116245 0 0) 0x7f2a4413a830 con 0x7f2a44070170 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): my global_id is 4542 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a4413acc0 con 0x7f2a44070170 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984645765 0 0) 0x7f2a441d5cc0 con 0x7f2a44070550 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 10 monclient(hunting): my global_id is 4628 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.041+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a4413a830 con 0x7f2a44070550 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2449729715 0 0) 0x7f2a4413acc0 con 0x7f2a44070170 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a24006680 con 0x7f2a44070170 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4059913621 0 0) 0x7f2a4413a830 con 0x7f2a44070550 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a4413acc0 con 0x7f2a44070550 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2a340016d0 con 0x7f2a44070170 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 10 monclient(hunting): dump: 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2a3c001af0 con 0x7f2a44070550 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 10 monclient(hunting): dump: 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.042+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3256669714 0 0) 0x7f2a24006680 con 0x7f2a44070170 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 >> v1:172.21.15.114:6790/0 conn(0x7f2a44070550 legacy=0x7f2a441d4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 >> v1:172.21.15.176:6789/0 conn(0x7f2a440713f0 legacy=0x7f2a441d55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 1 monclient: found mon.a 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a441336b0 con 0x7f2a44070170 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 10 monclient: _finish_auth 0 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a4a932640 5 monclient: authenticate success, global_id 4542 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a4a932640 10 monclient: _renew_subs 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a4a932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a4a932640 1 -- 172.21.15.114:0/3824310386 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a44139d10 con 0x7f2a44070170 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a4a932640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a4a932640 10 monclient: _renew_subs 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a4a932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a4a932640 1 -- 172.21.15.114:0/3824310386 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a44132df0 con 0x7f2a44070170 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a4a932640 1 librados: init done 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a34001be0 con 0x7f2a44070170 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2a34004a00 con 0x7f2a44070170 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.045+0000 7f2a40ff9640 10 monclient: dump: 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:14.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.046+0000 7f2a427fc640 4 set_mon_vals no callback set 2024-03-29T10:48:14.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.047+0000 7f2a4a932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:14.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.047+0000 7f2a4a932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:14.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.047+0000 7f2a4a932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:14.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.047+0000 7f2a4a932640 1 -- 172.21.15.114:0/3824310386 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a28005080 con 0x7f2a44070170 2024-03-29T10:48:14.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.047+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f2a3401a9d0 con 0x7f2a44070170 2024-03-29T10:48:14.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.048+0000 7f2a437fe640 10 monclient: get_auth_request con 0x7f2a24072340 auth_method 0 2024-03-29T10:48:14.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.048+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f2a34084960 con 0x7f2a44070170 2024-03-29T10:48:14.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.048+0000 7f2a40ff9640 10 client.4542.objecter ms_dispatch 0x7f2a44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:14.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.048+0000 7f2a40ff9640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:14.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.048+0000 7f2a40ff9640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:14.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.048+0000 7f2a40ff9640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T10:48:14.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.048+0000 7f2a40ff9640 10 client.4542.objecter ms_handle_connect 0x7f2a24072340 2024-03-29T10:48:14.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.057+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a34050240 con 0x7f2a44070170 2024-03-29T10:48:14.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.057+0000 7f2a40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:14.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.057+0000 7f2a40ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:14.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.386+0000 7f2a4a932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:14.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.386+0000 7f2a4a932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:14.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.386+0000 7f2a4a932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:14.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.386+0000 7f2a4a932640 1 -- 172.21.15.114:0/3824310386 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a28005330 con 0x7f2a44070170 2024-03-29T10:48:14.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.387+0000 7f2a40ff9640 1 -- 172.21.15.114:0/3824310386 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f2a34052170 con 0x7f2a44070170 2024-03-29T10:48:14.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.387+0000 7f2a40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:14.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.387+0000 7f2a40ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:14.372 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:48:14.372 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:14.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.392+0000 7f2a4a932640 10 librados: watch_flush enter 2024-03-29T10:48:14.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.392+0000 7f2a4a932640 10 librados: watch_flush exit 2024-03-29T10:48:14.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.392+0000 7f2a4a932640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T10:48:14.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.392+0000 7f2a4a932640 10 client.4542.objecter successfully canceled tick 2024-03-29T10:48:14.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.393+0000 7f2a4a932640 1 -- 172.21.15.114:0/3824310386 >> v1:172.21.15.114:6816/775044307 conn(0x7f2a24072340 legacy=0x7f2a24074800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:14.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.393+0000 7f2a4a932640 10 monclient: shutdown 2024-03-29T10:48:14.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.393+0000 7f2a4a932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:14.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.393+0000 7f2a4a932640 1 -- 172.21.15.114:0/3824310386 >> v1:172.21.15.114:6789/0 conn(0x7f2a44070170 legacy=0x7f2a441d1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:14.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.394+0000 7f2a4a932640 1 -- 172.21.15.114:0/3824310386 shutdown_connections 2024-03-29T10:48:14.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.394+0000 7f2a4a932640 1 -- 172.21.15.114:0/3824310386 >> 172.21.15.114:0/3824310386 conn(0x7f2a44067610 msgr2=0x7f2a4413d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:14.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.394+0000 7f2a4a932640 1 -- 172.21.15.114:0/3824310386 shutdown_connections 2024-03-29T10:48:14.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.394+0000 7f2a4a932640 1 -- 172.21.15.114:0/3824310386 wait complete. 2024-03-29T10:48:14.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:14.394+0000 7f2a4a932640 1 librados: shutdown 2024-03-29T10:48:14.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:21.203 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:21.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.366+0000 7f4d54cb3640 10 monclient: get_monmap_and_config 2024-03-29T10:48:21.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.366+0000 7f4d54cb3640 10 monclient: build_initial_monmap 2024-03-29T10:48:21.349 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.366+0000 7f4d54cb3640 10 monclient: monmap: 2024-03-29T10:48:21.349 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:21.349 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:21.349 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:48:21.366912+0000 2024-03-29T10:48:21.349 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:48:21.366912+0000 2024-03-29T10:48:21.349 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:21.349 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:21.349 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:21.349 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:48:21.349 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:48:21.349 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:21.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.369+0000 7f4d54cb3640 1 Processor -- start 2024-03-29T10:48:21.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.369+0000 7f4d54cb3640 1 -- start start 2024-03-29T10:48:21.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.369+0000 7f4d54cb3640 10 monclient: init 2024-03-29T10:48:21.355 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.370+0000 7f4d54cb3640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:21.355 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.370+0000 7f4d54cb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:21.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.371+0000 7f4d54cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f4d500b0860 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:21.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.371+0000 7f4d54cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f4d5013ef30 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:21.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.371+0000 7f4d54cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f4d500af8e0 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:21.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.371+0000 7f4d54cb3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d500b0c40 con 0x7f4d500b0860 2024-03-29T10:48:21.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.371+0000 7f4d54cb3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d500aed10 con 0x7f4d5013ef30 2024-03-29T10:48:21.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.371+0000 7f4d54cb3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d500afd50 con 0x7f4d500af8e0 2024-03-29T10:48:21.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.371+0000 7f4d54cb3640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:21.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.371+0000 7f4d54cb3640 10 monclient(hunting): authenticate will time out at 1366.611124s 2024-03-29T10:48:21.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4e59c640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4d500b0860 0x7f4d5013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49809/0 (socket says 172.21.15.114:49809) 2024-03-29T10:48:21.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4e59c640 1 -- 172.21.15.114:0/2816735205 learned_addr learned my addr 172.21.15.114:0/2816735205 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:21.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 1 -- 172.21.15.114:0/2816735205 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929403583 0 0) 0x7f4d500b0c40 con 0x7f4d500b0860 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 10 monclient(hunting): my global_id is 4545 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 1 -- 172.21.15.114:0/2816735205 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d38003500 con 0x7f4d500b0860 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 1 -- 172.21.15.114:0/2816735205 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3987689451 0 0) 0x7f4d38003500 con 0x7f4d500b0860 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 1 -- 172.21.15.114:0/2816735205 >> v1:172.21.15.114:6790/0 conn(0x7f4d500af8e0 legacy=0x7f4d50149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 1 -- 172.21.15.114:0/2816735205 >> v1:172.21.15.176:6789/0 conn(0x7f4d5013ef30 legacy=0x7f4d500af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 1 monclient: found mon.noname-a 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:48:21.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 1 -- 172.21.15.114:0/2816735205 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500ab1b0 con 0x7f4d500b0860 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 10 monclient: _finish_auth 0 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d54cb3640 5 monclient: authenticate success, global_id 4545 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d54cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 1 -- 172.21.15.114:0/2816735205 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4d40004cb0 con 0x7f4d500b0860 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d4d59a640 10 monclient: dump: 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:21.359 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.374+0000 7f4d54cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.375+0000 7f4d4d59a640 1 -- 172.21.15.114:0/2816735205 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d40004f40 con 0x7f4d500b0860 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.375+0000 7f4d4d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.375+0000 7f4d54cb3640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.375+0000 7f4d54cb3640 4 set_mon_vals no callback set 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.375+0000 7f4d4d59a640 1 -- 172.21.15.114:0/2816735205 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4d40001470 con 0x7f4d500b0860 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.375+0000 7f4d4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:21.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.375+0000 7f4d4d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:21.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.375+0000 7f4d4d59a640 10 monclient: dump: 2024-03-29T10:48:21.361 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:21.361 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:21.361 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:21.361 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:21.361 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:21.361 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:21.361 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:21.361 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:21.361 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:21.361 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:21.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.377+0000 7f4d54cb3640 10 monclient: shutdown 2024-03-29T10:48:21.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.377+0000 7f4d54cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:21.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.377+0000 7f4d54cb3640 1 -- 172.21.15.114:0/2816735205 >> v1:172.21.15.114:6789/0 conn(0x7f4d500b0860 legacy=0x7f4d5013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.377+0000 7f4d54cb3640 1 -- 172.21.15.114:0/2816735205 shutdown_connections 2024-03-29T10:48:21.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.377+0000 7f4d54cb3640 1 -- 172.21.15.114:0/2816735205 >> 172.21.15.114:0/2816735205 conn(0x7f4d5006b660 msgr2=0x7f4d5006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:21.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.379+0000 7f4d54cb3640 1 -- 172.21.15.114:0/2816735205 shutdown_connections 2024-03-29T10:48:21.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.379+0000 7f4d54cb3640 1 -- 172.21.15.114:0/2816735205 wait complete. 2024-03-29T10:48:21.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.380+0000 7f4d54cb3640 10 monclient: build_initial_monmap 2024-03-29T10:48:21.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.380+0000 7f4d54cb3640 10 monclient: monmap: 2024-03-29T10:48:21.363 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:21.363 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:21.364 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:48:21.364 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:48:21.364 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:21.364 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:21.364 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:21.364 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:48:21.364 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:48:21.364 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:21.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.380+0000 7f4d54cb3640 1 librados: starting msgr at 2024-03-29T10:48:21.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.380+0000 7f4d54cb3640 1 librados: starting objecter 2024-03-29T10:48:21.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.380+0000 7f4d54cb3640 1 Processor -- start 2024-03-29T10:48:21.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.382+0000 7f4d54cb3640 1 -- start start 2024-03-29T10:48:21.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.382+0000 7f4d54cb3640 1 librados: setting wanted keys 2024-03-29T10:48:21.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.382+0000 7f4d54cb3640 1 librados: calling monclient init 2024-03-29T10:48:21.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.382+0000 7f4d54cb3640 10 monclient: init 2024-03-29T10:48:21.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.383+0000 7f4d54cb3640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:21.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.383+0000 7f4d54cb3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:21.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.383+0000 7f4d54cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f4d500af8e0 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:21.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.383+0000 7f4d54cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f4d500b0860 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:21.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.383+0000 7f4d54cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f4d5013ef30 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:21.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.383+0000 7f4d54cb3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d501d3d30 con 0x7f4d500af8e0 2024-03-29T10:48:21.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.383+0000 7f4d54cb3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d500b43f0 con 0x7f4d5013ef30 2024-03-29T10:48:21.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.383+0000 7f4d54cb3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d501d3910 con 0x7f4d500b0860 2024-03-29T10:48:21.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.383+0000 7f4d54cb3640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:21.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.383+0000 7f4d54cb3640 10 monclient(hunting): authenticate will time out at 1366.623922s 2024-03-29T10:48:21.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d4dd9b640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f4d500b0860 0x7f4d50135580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:46522/0 (socket says 172.21.15.114:46522) 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d4dd9b640 1 -- 172.21.15.114:0/1670824181 learned_addr learned my addr 172.21.15.114:0/1670824181 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 10 client.?.objecter ms_handle_connect 0x7f4d500b0860 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413336731 0 0) 0x7f4d501d3910 con 0x7f4d500b0860 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 10 monclient(hunting): my global_id is 4631 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30004eb0 con 0x7f4d500b0860 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1971030437 0 0) 0x7f4d30004eb0 con 0x7f4d500b0860 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d501d3910 con 0x7f4d500b0860 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4d44003fe0 con 0x7f4d500b0860 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:21.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 10 monclient(hunting): dump: 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.384+0000 7f4d3effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.385+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 821674236 0 0) 0x7f4d501d3910 con 0x7f4d500b0860 2024-03-29T10:48:21.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.385+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 >> v1:172.21.15.176:6789/0 conn(0x7f4d5013ef30 legacy=0x7f4d501ddd70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.387+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 >> v1:172.21.15.114:6789/0 conn(0x7f4d500af8e0 legacy=0x7f4d501d3200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 1 monclient: found mon.c 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500b3780 con 0x7f4d500b0860 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 10 monclient: _finish_auth 0 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d54cb3640 5 monclient: authenticate success, global_id 4631 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d54cb3640 10 monclient: _renew_subs 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d54cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d54cb3640 1 -- 172.21.15.114:0/1670824181 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d500b3a30 con 0x7f4d500b0860 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d54cb3640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:21.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d54cb3640 10 monclient: _renew_subs 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d54cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d54cb3640 1 -- 172.21.15.114:0/1670824181 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d50135e70 con 0x7f4d500b0860 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d54cb3640 1 librados: init done 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d44000ed0 con 0x7f4d500b0860 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f4d440012a0 con 0x7f4d500b0860 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.388+0000 7f4d3effd640 10 monclient: dump: 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:21.376 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:21.377 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:21.377 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:21.377 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:21.377 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:21.377 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:21.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.389+0000 7f4d4cd99640 4 set_mon_vals no callback set 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.391+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f4d44001480 con 0x7f4d500b0860 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.392+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f4d440848e0 con 0x7f4d500b0860 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.392+0000 7f4d3effd640 10 client.4631.objecter ms_dispatch 0x7f4d50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.392+0000 7f4d3effd640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.392+0000 7f4d3effd640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.392+0000 7f4d54cb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.392+0000 7f4d54cb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.392+0000 7f4d54cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.392+0000 7f4d3effd640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.392+0000 7f4d54cb3640 1 -- 172.21.15.114:0/1670824181 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d34005080 con 0x7f4d500b0860 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.393+0000 7f4d4e59c640 10 monclient: get_auth_request con 0x7f4d3006dc60 auth_method 0 2024-03-29T10:48:21.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.393+0000 7f4d3effd640 10 client.4631.objecter ms_handle_connect 0x7f4d3006dc60 2024-03-29T10:48:21.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.401+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d44050220 con 0x7f4d500b0860 2024-03-29T10:48:21.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.401+0000 7f4d3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:21.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.401+0000 7f4d3effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:21.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.730+0000 7f4d54cb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:21.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.730+0000 7f4d54cb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:21.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.730+0000 7f4d54cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:48:21.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.730+0000 7f4d54cb3640 1 -- 172.21.15.114:0/1670824181 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d34005330 con 0x7f4d500b0860 2024-03-29T10:48:21.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.731+0000 7f4d3effd640 1 -- 172.21.15.114:0/1670824181 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f4d44050220 con 0x7f4d500b0860 2024-03-29T10:48:21.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.731+0000 7f4d3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:21.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.731+0000 7f4d3effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:21.713 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:48:21.713 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:21.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.734+0000 7f4d54cb3640 10 librados: watch_flush enter 2024-03-29T10:48:21.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.734+0000 7f4d54cb3640 10 librados: watch_flush exit 2024-03-29T10:48:21.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.734+0000 7f4d54cb3640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T10:48:21.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.734+0000 7f4d54cb3640 10 client.4631.objecter successfully canceled tick 2024-03-29T10:48:21.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.734+0000 7f4d54cb3640 1 -- 172.21.15.114:0/1670824181 >> v1:172.21.15.114:6816/775044307 conn(0x7f4d3006dc60 legacy=0x7f4d30070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.734+0000 7f4d54cb3640 10 monclient: shutdown 2024-03-29T10:48:21.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.734+0000 7f4d54cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:21.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.734+0000 7f4d54cb3640 1 -- 172.21.15.114:0/1670824181 >> v1:172.21.15.114:6790/0 conn(0x7f4d500b0860 legacy=0x7f4d50135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.735+0000 7f4d54cb3640 1 -- 172.21.15.114:0/1670824181 shutdown_connections 2024-03-29T10:48:21.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.735+0000 7f4d54cb3640 1 -- 172.21.15.114:0/1670824181 >> 172.21.15.114:0/1670824181 conn(0x7f4d5006b660 msgr2=0x7f4d5013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:21.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.735+0000 7f4d54cb3640 1 -- 172.21.15.114:0/1670824181 shutdown_connections 2024-03-29T10:48:21.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.735+0000 7f4d54cb3640 1 -- 172.21.15.114:0/1670824181 wait complete. 2024-03-29T10:48:21.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:21.736+0000 7f4d54cb3640 1 librados: shutdown 2024-03-29T10:48:21.734 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:28.542 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.707+0000 7f37b1a4e640 10 monclient: get_monmap_and_config 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.707+0000 7f37b1a4e640 10 monclient: build_initial_monmap 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.707+0000 7f37b1a4e640 10 monclient: monmap: 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:48:28.708697+0000 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:48:28.708697+0000 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.709+0000 7f37b1a4e640 1 Processor -- start 2024-03-29T10:48:28.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.709+0000 7f37b1a4e640 1 -- start start 2024-03-29T10:48:28.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.709+0000 7f37b1a4e640 10 monclient: init 2024-03-29T10:48:28.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.710+0000 7f37b1a4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:28.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.710+0000 7f37b1a4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:28.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.710+0000 7f37b1a4e640 10 monclient(hunting): picked mon.noname-a con 0x7f37ac0b0770 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:28.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.710+0000 7f37b1a4e640 10 monclient(hunting): picked mon.noname-c con 0x7f37ac0ad850 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:28.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.710+0000 7f37b1a4e640 10 monclient(hunting): picked mon.noname-b con 0x7f37ac13c1b0 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:28.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.710+0000 7f37b1a4e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0ac8e0 con 0x7f37ac0b0770 2024-03-29T10:48:28.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.710+0000 7f37b1a4e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0ad330 con 0x7f37ac0ad850 2024-03-29T10:48:28.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.710+0000 7f37b1a4e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0acfe0 con 0x7f37ac13c1b0 2024-03-29T10:48:28.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.710+0000 7f37b1a4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:28.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.710+0000 7f37b1a4e640 10 monclient(hunting): authenticate will time out at 1373.951710s 2024-03-29T10:48:28.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.710+0000 7f37ab7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f37ac13c1b0 0x7f37ac14b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:46892/0 (socket says 172.21.15.114:46892) 2024-03-29T10:48:28.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.711+0000 7f37ab7fe640 1 -- 172.21.15.114:0/1556703699 learned_addr learned my addr 172.21.15.114:0/1556703699 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:28.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.714+0000 7f37a9ffb640 1 -- 172.21.15.114:0/1556703699 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031534374 0 0) 0x7f37ac0acfe0 con 0x7f37ac13c1b0 2024-03-29T10:48:28.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.714+0000 7f37a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:28.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.714+0000 7f37a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:28.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.714+0000 7f37a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:28.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.714+0000 7f37a9ffb640 10 monclient(hunting): my global_id is 4634 2024-03-29T10:48:28.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.714+0000 7f37a9ffb640 1 -- 172.21.15.114:0/1556703699 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3794003520 con 0x7f37ac13c1b0 2024-03-29T10:48:28.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.714+0000 7f37a9ffb640 1 -- 172.21.15.114:0/1556703699 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901264251 0 0) 0x7f37ac0ac8e0 con 0x7f37ac0b0770 2024-03-29T10:48:28.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:28.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:28.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:28.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 10 monclient(hunting): my global_id is 4548 2024-03-29T10:48:28.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 1 -- 172.21.15.114:0/1556703699 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ac0acfe0 con 0x7f37ac0b0770 2024-03-29T10:48:28.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 1 -- 172.21.15.114:0/1556703699 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 826148874 0 0) 0x7f37ac0acfe0 con 0x7f37ac0b0770 2024-03-29T10:48:28.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 1 -- 172.21.15.114:0/1556703699 >> v1:172.21.15.114:6790/0 conn(0x7f37ac13c1b0 legacy=0x7f37ac14b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:28.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 1 -- 172.21.15.114:0/1556703699 >> v1:172.21.15.176:6789/0 conn(0x7f37ac0ad850 legacy=0x7f37ac0aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:28.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:28.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 1 monclient: found mon.noname-a 2024-03-29T10:48:28.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:28.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 1 -- 172.21.15.114:0/1556703699 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac0ab1f0 con 0x7f37ac0b0770 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 1 -- 172.21.15.114:0/1556703699 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f37a40019f0 con 0x7f37ac0b0770 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 10 monclient: dump: 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:28.700 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:28.701 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:28.701 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:28.701 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:28.701 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:28.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37b1a4e640 5 monclient: authenticate success, global_id 4548 2024-03-29T10:48:28.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37b1a4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:28.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.715+0000 7f37a9ffb640 1 -- 172.21.15.114:0/1556703699 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37a4001b50 con 0x7f37ac0b0770 2024-03-29T10:48:28.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.716+0000 7f37a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:28.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.716+0000 7f37b1a4e640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:28.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.716+0000 7f37b1a4e640 4 set_mon_vals no callback set 2024-03-29T10:48:28.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.716+0000 7f37b1a4e640 10 monclient: shutdown 2024-03-29T10:48:28.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.716+0000 7f37b1a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:28.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.716+0000 7f37b1a4e640 1 -- 172.21.15.114:0/1556703699 >> v1:172.21.15.114:6789/0 conn(0x7f37ac0b0770 legacy=0x7f37ac072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:28.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.717+0000 7f37b1a4e640 1 -- 172.21.15.114:0/1556703699 shutdown_connections 2024-03-29T10:48:28.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.717+0000 7f37b1a4e640 1 -- 172.21.15.114:0/1556703699 >> 172.21.15.114:0/1556703699 conn(0x7f37ac06b660 msgr2=0x7f37ac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:28.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.717+0000 7f37b1a4e640 1 -- 172.21.15.114:0/1556703699 shutdown_connections 2024-03-29T10:48:28.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.717+0000 7f37b1a4e640 1 -- 172.21.15.114:0/1556703699 wait complete. 2024-03-29T10:48:28.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.718+0000 7f37b1a4e640 10 monclient: build_initial_monmap 2024-03-29T10:48:28.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.718+0000 7f37b1a4e640 10 monclient: monmap: 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.718+0000 7f37b1a4e640 1 librados: starting msgr at 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.718+0000 7f37b1a4e640 1 librados: starting objecter 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.719+0000 7f37b1a4e640 1 Processor -- start 2024-03-29T10:48:28.713 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.719+0000 7f37b1a4e640 1 -- start start 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.719+0000 7f37b1a4e640 1 librados: setting wanted keys 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.719+0000 7f37b1a4e640 1 librados: calling monclient init 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.719+0000 7f37b1a4e640 10 monclient: init 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.720+0000 7f37b1a4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.720+0000 7f37b1a4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.720+0000 7f37b1a4e640 10 monclient(hunting): picked mon.noname-c con 0x7f37ac0ad850 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.720+0000 7f37b1a4e640 10 monclient(hunting): picked mon.noname-a con 0x7f37ac0b0770 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.720+0000 7f37b1a4e640 10 monclient(hunting): picked mon.noname-b con 0x7f37ac13c1b0 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.720+0000 7f37b1a4e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac070de0 con 0x7f37ac0b0770 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.720+0000 7f37b1a4e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac070a40 con 0x7f37ac13c1b0 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.720+0000 7f37b1a4e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac06f4a0 con 0x7f37ac0ad850 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.720+0000 7f37b1a4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.720+0000 7f37b1a4e640 10 monclient(hunting): authenticate will time out at 1373.961166s 2024-03-29T10:48:28.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.721+0000 7f37aaffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f37ac0ad850 0x7f37ac071300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:57912/0 (socket says 172.21.15.114:57912) 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.721+0000 7f37aaffd640 1 -- 172.21.15.114:0/3197692923 learned_addr learned my addr 172.21.15.114:0/3197692923 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.721+0000 7f3793fff640 10 client.?.objecter ms_handle_connect 0x7f37ac0ad850 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.721+0000 7f3793fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.721+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601574269 0 0) 0x7f37ac06f4a0 con 0x7f37ac0ad850 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.722+0000 7f3793fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.722+0000 7f3793fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.722+0000 7f3793fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.722+0000 7f3793fff640 10 monclient(hunting): my global_id is 4637 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.722+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3798004eb0 con 0x7f37ac0ad850 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.722+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2400236467 0 0) 0x7f3798004eb0 con 0x7f37ac0ad850 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.722+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37ac06f4a0 con 0x7f37ac0ad850 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.722+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f37a4003b10 con 0x7f37ac0ad850 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.722+0000 7f3793fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:28.715 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.723+0000 7f3793fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.723+0000 7f3793fff640 10 monclient(hunting): dump: 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.723+0000 7f3793fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.723+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3237494844 0 0) 0x7f37ac06f4a0 con 0x7f37ac0ad850 2024-03-29T10:48:28.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.723+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 >> v1:172.21.15.176:6789/0 conn(0x7f37ac13c1b0 legacy=0x7f37ac06ed90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 >> v1:172.21.15.114:6789/0 conn(0x7f37ac0b0770 legacy=0x7f37ac0b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f3793fff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f3793fff640 1 monclient: found mon.c 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f3793fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f3793fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac06f630 con 0x7f37ac0ad850 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f3793fff640 10 monclient: _finish_auth 0 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f3793fff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f3793fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f37b1a4e640 5 monclient: authenticate success, global_id 4637 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f37b1a4e640 10 monclient: _renew_subs 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f37b1a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.725+0000 7f37b1a4e640 1 -- 172.21.15.114:0/3197692923 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac06e4f0 con 0x7f37ac0ad850 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.726+0000 7f37b1a4e640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:28.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.726+0000 7f37b1a4e640 10 monclient: _renew_subs 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.726+0000 7f37b1a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.726+0000 7f37b1a4e640 1 -- 172.21.15.114:0/3197692923 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ac135a00 con 0x7f37ac0ad850 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.726+0000 7f37b1a4e640 1 librados: init done 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.726+0000 7f3793fff640 10 client.4637.objecter ms_handle_connect 0x7f37ac0b0770 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.726+0000 7f3793fff640 10 client.4637.objecter resend_mon_ops 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.726+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37a4004080 con 0x7f37ac0ad850 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.726+0000 7f3793fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.726+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f37a4004490 con 0x7f37ac0ad850 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.726+0000 7f37a97fa640 4 set_mon_vals no callback set 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.729+0000 7f3793fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.729+0000 7f3793fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.729+0000 7f3793fff640 10 monclient: dump: 2024-03-29T10:48:28.718 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.729+0000 7f37b1a4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.729+0000 7f37b1a4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.729+0000 7f37b1a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.729+0000 7f37b1a4e640 1 -- 172.21.15.114:0/3197692923 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378c005080 con 0x7f37ac0ad850 2024-03-29T10:48:28.719 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.730+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f37a4001dc0 con 0x7f37ac0ad850 2024-03-29T10:48:28.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.730+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f37a4085010 con 0x7f37ac0ad850 2024-03-29T10:48:28.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.730+0000 7f3793fff640 10 client.4637.objecter ms_dispatch 0x7f37ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:28.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.730+0000 7f3793fff640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:28.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.730+0000 7f3793fff640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:28.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.730+0000 7f3793fff640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T10:48:28.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.731+0000 7f37aa7fc640 10 monclient: get_auth_request con 0x7f379806db80 auth_method 0 2024-03-29T10:48:28.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.732+0000 7f3793fff640 10 client.4637.objecter ms_handle_connect 0x7f379806db80 2024-03-29T10:48:28.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.740+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37a40508f0 con 0x7f37ac0ad850 2024-03-29T10:48:28.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.740+0000 7f3793fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:28.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:28.740+0000 7f3793fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:29.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.073+0000 7f37b1a4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:29.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.073+0000 7f37b1a4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:29.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.073+0000 7f37b1a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:48:29.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.073+0000 7f37b1a4e640 1 -- 172.21.15.114:0/3197692923 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f378c005330 con 0x7f37ac0ad850 2024-03-29T10:48:29.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.074+0000 7f3793fff640 1 -- 172.21.15.114:0/3197692923 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f37a4052820 con 0x7f37ac0ad850 2024-03-29T10:48:29.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.074+0000 7f3793fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:29.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.074+0000 7f3793fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:29.057 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:48:29.057 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:29.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.078+0000 7f37b1a4e640 10 librados: watch_flush enter 2024-03-29T10:48:29.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.078+0000 7f37b1a4e640 10 librados: watch_flush exit 2024-03-29T10:48:29.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.078+0000 7f37b1a4e640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T10:48:29.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.078+0000 7f37b1a4e640 10 client.4637.objecter successfully canceled tick 2024-03-29T10:48:29.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.079+0000 7f37b1a4e640 1 -- 172.21.15.114:0/3197692923 >> v1:172.21.15.114:6816/775044307 conn(0x7f379806db80 legacy=0x7f3798070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:29.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.079+0000 7f37b1a4e640 10 monclient: shutdown 2024-03-29T10:48:29.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.079+0000 7f37b1a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:29.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.079+0000 7f37b1a4e640 1 -- 172.21.15.114:0/3197692923 >> v1:172.21.15.114:6790/0 conn(0x7f37ac0ad850 legacy=0x7f37ac071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:29.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.079+0000 7f37b1a4e640 1 -- 172.21.15.114:0/3197692923 shutdown_connections 2024-03-29T10:48:29.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.079+0000 7f37b1a4e640 1 -- 172.21.15.114:0/3197692923 >> 172.21.15.114:0/3197692923 conn(0x7f37ac06b660 msgr2=0x7f37ac0aa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:29.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.079+0000 7f37b1a4e640 1 -- 172.21.15.114:0/3197692923 shutdown_connections 2024-03-29T10:48:29.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.079+0000 7f37b1a4e640 1 -- 172.21.15.114:0/3197692923 wait complete. 2024-03-29T10:48:29.062 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:29.079+0000 7f37b1a4e640 1 librados: shutdown 2024-03-29T10:48:29.079 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:35.886 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:36.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.053+0000 7f5308d2b640 10 monclient: get_monmap_and_config 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.053+0000 7f5308d2b640 10 monclient: build_initial_monmap 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.053+0000 7f5308d2b640 10 monclient: monmap: 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:48:36.054715+0000 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:48:36.054715+0000 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:48:36.036 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:36.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.055+0000 7f5308d2b640 1 Processor -- start 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.056+0000 7f5308d2b640 1 -- start start 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.056+0000 7f5308d2b640 10 monclient: init 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.057+0000 7f5308d2b640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.057+0000 7f5308d2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.058+0000 7f5308d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f53041457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.058+0000 7f5308d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f530413f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.058+0000 7f5308d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f53040a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.058+0000 7f5308d2b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53040aaef0 con 0x7f530413f650 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.059+0000 7f5308d2b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5304145bd0 con 0x7f53040a9d70 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.059+0000 7f5308d2b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5304145eb0 con 0x7f53041457f0 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.059+0000 7f5308d2b640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.059+0000 7f5308d2b640 10 monclient(hunting): authenticate will time out at 1381.300045s 2024-03-29T10:48:36.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.060+0000 7f530259c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f53041457f0 0x7f530413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:48863/0 (socket says 172.21.15.114:48863) 2024-03-29T10:48:36.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.060+0000 7f530259c640 1 -- 172.21.15.114:0/300518765 learned_addr learned my addr 172.21.15.114:0/300518765 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:36.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.061+0000 7f530159a640 1 -- 172.21.15.114:0/300518765 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971139618 0 0) 0x7f5304145eb0 con 0x7f53041457f0 2024-03-29T10:48:36.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.061+0000 7f530159a640 10 monclient(hunting): _negotiate 2024-03-29T10:48:36.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.061+0000 7f530159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:36.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.061+0000 7f530159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:36.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.061+0000 7f530159a640 10 monclient(hunting): my global_id is 4640 2024-03-29T10:48:36.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.061+0000 7f530159a640 1 -- 172.21.15.114:0/300518765 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ec003520 con 0x7f53041457f0 2024-03-29T10:48:36.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.062+0000 7f530159a640 1 -- 172.21.15.114:0/300518765 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 341955646 0 0) 0x7f52ec003520 con 0x7f53041457f0 2024-03-29T10:48:36.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.062+0000 7f530159a640 1 -- 172.21.15.114:0/300518765 >> v1:172.21.15.176:6789/0 conn(0x7f53040a9d70 legacy=0x7f530414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:36.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 1 -- 172.21.15.114:0/300518765 >> v1:172.21.15.114:6789/0 conn(0x7f530413f650 legacy=0x7f53040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 10 monclient: _finish_hunting 0 2024-03-29T10:48:36.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 1 monclient: found mon.noname-b 2024-03-29T10:48:36.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 1 -- 172.21.15.114:0/300518765 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5304146340 con 0x7f53041457f0 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 10 monclient: _finish_auth 0 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 10 monclient: _check_auth_tickets 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 1 -- 172.21.15.114:0/300518765 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f52f4001690 con 0x7f53041457f0 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 10 monclient: dump: 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:36.050 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 1 -- 172.21.15.114:0/300518765 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52f4001860 con 0x7f53041457f0 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 1 -- 172.21.15.114:0/300518765 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f52f4001cb0 con 0x7f53041457f0 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.066+0000 7f530159a640 10 monclient: dump: 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:36.051 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.067+0000 7f5308d2b640 5 monclient: authenticate success, global_id 4640 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.067+0000 7f5308d2b640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.067+0000 7f5308d2b640 4 set_mon_vals no callback set 2024-03-29T10:48:36.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.068+0000 7f5308d2b640 10 monclient: shutdown 2024-03-29T10:48:36.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.068+0000 7f5308d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:36.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.068+0000 7f5308d2b640 1 -- 172.21.15.114:0/300518765 >> v1:172.21.15.114:6790/0 conn(0x7f53041457f0 legacy=0x7f530413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.068+0000 7f5308d2b640 1 -- 172.21.15.114:0/300518765 shutdown_connections 2024-03-29T10:48:36.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.068+0000 7f5308d2b640 1 -- 172.21.15.114:0/300518765 >> 172.21.15.114:0/300518765 conn(0x7f5304066a00 msgr2=0x7f5304066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:36.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.071+0000 7f5308d2b640 1 -- 172.21.15.114:0/300518765 shutdown_connections 2024-03-29T10:48:36.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.072+0000 7f5308d2b640 1 -- 172.21.15.114:0/300518765 wait complete. 2024-03-29T10:48:36.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.073+0000 7f5308d2b640 10 monclient: build_initial_monmap 2024-03-29T10:48:36.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.073+0000 7f5308d2b640 10 monclient: monmap: 2024-03-29T10:48:36.055 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:36.055 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:36.056 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:48:36.056 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:48:36.056 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:36.056 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:36.056 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:36.056 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:48:36.056 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:48:36.056 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:36.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.074+0000 7f5308d2b640 1 librados: starting msgr at 2024-03-29T10:48:36.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.074+0000 7f5308d2b640 1 librados: starting objecter 2024-03-29T10:48:36.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.074+0000 7f5308d2b640 1 Processor -- start 2024-03-29T10:48:36.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.074+0000 7f5308d2b640 1 -- start start 2024-03-29T10:48:36.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.074+0000 7f5308d2b640 1 librados: setting wanted keys 2024-03-29T10:48:36.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.074+0000 7f5308d2b640 1 librados: calling monclient init 2024-03-29T10:48:36.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.074+0000 7f5308d2b640 10 monclient: init 2024-03-29T10:48:36.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.075+0000 7f5308d2b640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:36.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.075+0000 7f5308d2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:36.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.075+0000 7f5308d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f53040a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:36.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.075+0000 7f5308d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f530413f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:36.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.079+0000 7f5308d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f53041457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:36.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.080+0000 7f5308d2b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5304135d10 con 0x7f530413f650 2024-03-29T10:48:36.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.080+0000 7f5308d2b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5304144730 con 0x7f53040a9d70 2024-03-29T10:48:36.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.080+0000 7f5308d2b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53041358a0 con 0x7f53041457f0 2024-03-29T10:48:36.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.080+0000 7f5308d2b640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:36.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.080+0000 7f5308d2b640 10 monclient(hunting): authenticate will time out at 1381.321753s 2024-03-29T10:48:36.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.080+0000 7f530259c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f53040a9d70 0x7f5304143410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:40321/0 (socket says 172.21.15.114:40321) 2024-03-29T10:48:36.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.080+0000 7f530259c640 1 -- 172.21.15.114:0/2022542901 learned_addr learned my addr 172.21.15.114:0/2022542901 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:36.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.081+0000 7f52f2ffd640 10 client.?.objecter ms_handle_connect 0x7f53040a9d70 2024-03-29T10:48:36.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.081+0000 7f52f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:36.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.083+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73154659 0 0) 0x7f5304144730 con 0x7f53040a9d70 2024-03-29T10:48:36.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.083+0000 7f52f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:36.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.083+0000 7f52f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:36.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.083+0000 7f52f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:36.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.083+0000 7f52f2ffd640 10 monclient(hunting): my global_id is 4516 2024-03-29T10:48:36.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.083+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e4004eb0 con 0x7f53040a9d70 2024-03-29T10:48:36.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.083+0000 7f52f2ffd640 10 client.?.objecter ms_handle_connect 0x7f530413f650 2024-03-29T10:48:36.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.083+0000 7f52f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2871570087 0 0) 0x7f52e4004eb0 con 0x7f53040a9d70 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5304144730 con 0x7f53040a9d70 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583909907 0 0) 0x7f5304135d10 con 0x7f530413f650 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 10 monclient(hunting): my global_id is 4557 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e4004eb0 con 0x7f530413f650 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f52f4001c30 con 0x7f53040a9d70 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:36.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 10 monclient(hunting): dump: 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.085+0000 7f52f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 695788218 0 0) 0x7f5304144730 con 0x7f53040a9d70 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 >> v1:172.21.15.114:6790/0 conn(0x7f53041457f0 legacy=0x7f53041dda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 >> v1:172.21.15.114:6789/0 conn(0x7f530413f650 legacy=0x7f530413d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 1 monclient: found mon.b 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5304135a30 con 0x7f53040a9d70 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52f4004b00 con 0x7f53040a9d70 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f52f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.086+0000 7f5300d99640 4 set_mon_vals no callback set 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f52f4004f10 con 0x7f53040a9d70 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f52f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:36.074 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f52f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f52f2ffd640 10 monclient: dump: 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f5308d2b640 5 monclient: authenticate success, global_id 4516 2024-03-29T10:48:36.075 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f5308d2b640 10 monclient: _renew_subs 2024-03-29T10:48:36.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f5308d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:36.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f5308d2b640 1 -- 172.21.15.114:0/2022542901 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53041352a0 con 0x7f53040a9d70 2024-03-29T10:48:36.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f5308d2b640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:36.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f5308d2b640 10 monclient: _renew_subs 2024-03-29T10:48:36.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f5308d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:36.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f5308d2b640 1 -- 172.21.15.114:0/2022542901 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5304144e30 con 0x7f53040a9d70 2024-03-29T10:48:36.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.087+0000 7f5308d2b640 1 librados: init done 2024-03-29T10:48:36.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.090+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f52f40050f0 con 0x7f53040a9d70 2024-03-29T10:48:36.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.090+0000 7f5308d2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:36.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.090+0000 7f5308d2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:36.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.090+0000 7f5308d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:36.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.090+0000 7f5308d2b640 1 -- 172.21.15.114:0/2022542901 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5304075f10 con 0x7f53040a9d70 2024-03-29T10:48:36.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.092+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f52f4084c60 con 0x7f53040a9d70 2024-03-29T10:48:36.078 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.092+0000 7f52f2ffd640 10 client.4516.objecter ms_dispatch 0x7f5304000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:36.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.092+0000 7f52f2ffd640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:36.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.092+0000 7f52f2ffd640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:36.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.092+0000 7f52f2ffd640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T10:48:36.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.092+0000 7f5301d9b640 10 monclient: get_auth_request con 0x7f52e406dd60 auth_method 0 2024-03-29T10:48:36.079 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.093+0000 7f52f2ffd640 10 client.4516.objecter ms_handle_connect 0x7f52e406dd60 2024-03-29T10:48:36.081 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.099+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52f4050540 con 0x7f53040a9d70 2024-03-29T10:48:36.082 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.099+0000 7f52f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:36.082 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.099+0000 7f52f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:36.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.411+0000 7f5308d2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:36.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.411+0000 7f5308d2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:36.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.411+0000 7f5308d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:36.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.411+0000 7f5308d2b640 1 -- 172.21.15.114:0/2022542901 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53040b3680 con 0x7f53040a9d70 2024-03-29T10:48:36.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.413+0000 7f52f2ffd640 1 -- 172.21.15.114:0/2022542901 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f52f4052470 con 0x7f53040a9d70 2024-03-29T10:48:36.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.413+0000 7f52f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:36.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.413+0000 7f52f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:36.396 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:48:36.396 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:36.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.418+0000 7f5308d2b640 10 librados: watch_flush enter 2024-03-29T10:48:36.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.419+0000 7f5308d2b640 10 librados: watch_flush exit 2024-03-29T10:48:36.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.419+0000 7f5308d2b640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T10:48:36.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.419+0000 7f5308d2b640 10 client.4516.objecter successfully canceled tick 2024-03-29T10:48:36.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.419+0000 7f5308d2b640 1 -- 172.21.15.114:0/2022542901 >> v1:172.21.15.114:6816/775044307 conn(0x7f52e406dd60 legacy=0x7f52e4070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.419+0000 7f5308d2b640 10 monclient: shutdown 2024-03-29T10:48:36.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.419+0000 7f5308d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:36.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.419+0000 7f5308d2b640 1 -- 172.21.15.114:0/2022542901 >> v1:172.21.15.176:6789/0 conn(0x7f53040a9d70 legacy=0x7f5304143410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.419+0000 7f5308d2b640 1 -- 172.21.15.114:0/2022542901 shutdown_connections 2024-03-29T10:48:36.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.419+0000 7f5308d2b640 1 -- 172.21.15.114:0/2022542901 >> 172.21.15.114:0/2022542901 conn(0x7f5304066a00 msgr2=0x7f530413a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:36.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.420+0000 7f5308d2b640 1 -- 172.21.15.114:0/2022542901 shutdown_connections 2024-03-29T10:48:36.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.420+0000 7f5308d2b640 1 -- 172.21.15.114:0/2022542901 wait complete. 2024-03-29T10:48:36.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:36.422+0000 7f5308d2b640 1 librados: shutdown 2024-03-29T10:48:36.423 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:43.231 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:43.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.405+0000 7f7b4deaf640 10 monclient: get_monmap_and_config 2024-03-29T10:48:43.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.405+0000 7f7b4deaf640 10 monclient: build_initial_monmap 2024-03-29T10:48:43.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.405+0000 7f7b4deaf640 10 monclient: monmap: 2024-03-29T10:48:43.388 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:43.388 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:43.388 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:48:43.406386+0000 2024-03-29T10:48:43.388 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:48:43.406386+0000 2024-03-29T10:48:43.388 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:43.388 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:43.389 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:43.389 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:48:43.389 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:48:43.389 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:43.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.406+0000 7f7b4deaf640 1 Processor -- start 2024-03-29T10:48:43.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.406+0000 7f7b4deaf640 1 -- start start 2024-03-29T10:48:43.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.406+0000 7f7b4deaf640 10 monclient: init 2024-03-29T10:48:43.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.407+0000 7f7b4deaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:43.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.407+0000 7f7b4deaf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:43.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.407+0000 7f7b4deaf640 10 monclient(hunting): picked mon.noname-c con 0x7f7b480a9930 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:43.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.407+0000 7f7b4deaf640 10 monclient(hunting): picked mon.noname-a con 0x7f7b48149dc0 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:43.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.407+0000 7f7b4deaf640 10 monclient(hunting): picked mon.noname-b con 0x7f7b4814ff40 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:43.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.408+0000 7f7b4deaf640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b480aae60 con 0x7f7b48149dc0 2024-03-29T10:48:43.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.408+0000 7f7b4deaf640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b480b4310 con 0x7f7b480a9930 2024-03-29T10:48:43.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.408+0000 7f7b4deaf640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b480afdd0 con 0x7f7b4814ff40 2024-03-29T10:48:43.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.408+0000 7f7b4deaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:43.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.408+0000 7f7b4deaf640 10 monclient(hunting): authenticate will time out at 1388.649238s 2024-03-29T10:48:43.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.410+0000 7f7b477fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f7b480a9930 0x7f7b48143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:39817/0 (socket says 172.21.15.114:39817) 2024-03-29T10:48:43.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.410+0000 7f7b477fe640 1 -- 172.21.15.114:0/348483764 learned_addr learned my addr 172.21.15.114:0/348483764 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:43.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.411+0000 7f7b467fc640 1 -- 172.21.15.114:0/348483764 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442410733 0 0) 0x7f7b480afdd0 con 0x7f7b4814ff40 2024-03-29T10:48:43.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.411+0000 7f7b467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:43.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.411+0000 7f7b467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:43.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.411+0000 7f7b467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:43.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.411+0000 7f7b467fc640 10 monclient(hunting): my global_id is 4643 2024-03-29T10:48:43.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.411+0000 7f7b467fc640 1 -- 172.21.15.114:0/348483764 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b30003520 con 0x7f7b4814ff40 2024-03-29T10:48:43.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 1 -- 172.21.15.114:0/348483764 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032822254 0 0) 0x7f7b480b4310 con 0x7f7b480a9930 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient(hunting): my global_id is 4519 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 1 -- 172.21.15.114:0/348483764 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b480afdd0 con 0x7f7b480a9930 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 1 -- 172.21.15.114:0/348483764 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1782045595 0 0) 0x7f7b30003520 con 0x7f7b4814ff40 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 1 -- 172.21.15.114:0/348483764 >> v1:172.21.15.176:6789/0 conn(0x7f7b480a9930 legacy=0x7f7b48143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 1 -- 172.21.15.114:0/348483764 >> v1:172.21.15.114:6789/0 conn(0x7f7b48149dc0 legacy=0x7f7b4814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 1 monclient: found mon.noname-b 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:43.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 1 -- 172.21.15.114:0/348483764 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4814a950 con 0x7f7b4814ff40 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 1 -- 172.21.15.114:0/348483764 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7b40001a10 con 0x7f7b4814ff40 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient: dump: 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.399 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 1 -- 172.21.15.114:0/348483764 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b40001c30 con 0x7f7b4814ff40 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 1 -- 172.21.15.114:0/348483764 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7b40001fc0 con 0x7f7b4814ff40 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.412+0000 7f7b467fc640 10 monclient: dump: 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:43.400 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.413+0000 7f7b4deaf640 5 monclient: authenticate success, global_id 4643 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.413+0000 7f7b4deaf640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.413+0000 7f7b4deaf640 4 set_mon_vals no callback set 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.414+0000 7f7b4deaf640 10 monclient: shutdown 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.414+0000 7f7b4deaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.414+0000 7f7b4deaf640 1 -- 172.21.15.114:0/348483764 >> v1:172.21.15.114:6790/0 conn(0x7f7b4814ff40 legacy=0x7f7b48152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.414+0000 7f7b4deaf640 1 -- 172.21.15.114:0/348483764 shutdown_connections 2024-03-29T10:48:43.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.414+0000 7f7b4deaf640 1 -- 172.21.15.114:0/348483764 >> 172.21.15.114:0/348483764 conn(0x7f7b480b35e0 msgr2=0x7f7b480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.416+0000 7f7b4deaf640 1 -- 172.21.15.114:0/348483764 shutdown_connections 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.417+0000 7f7b4deaf640 1 -- 172.21.15.114:0/348483764 wait complete. 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.417+0000 7f7b4deaf640 10 monclient: build_initial_monmap 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.417+0000 7f7b4deaf640 10 monclient: monmap: 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.418+0000 7f7b4deaf640 1 librados: starting msgr at 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.418+0000 7f7b4deaf640 1 librados: starting objecter 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.418+0000 7f7b4deaf640 1 Processor -- start 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.418+0000 7f7b4deaf640 1 -- start start 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.418+0000 7f7b4deaf640 1 librados: setting wanted keys 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.418+0000 7f7b4deaf640 1 librados: calling monclient init 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.418+0000 7f7b4deaf640 10 monclient: init 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.419+0000 7f7b4deaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.419+0000 7f7b4deaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:43.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.422+0000 7f7b4deaf640 10 monclient(hunting): picked mon.noname-a con 0x7f7b480a9930 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.423+0000 7f7b4deaf640 10 monclient(hunting): picked mon.noname-b con 0x7f7b48149dc0 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.423+0000 7f7b4deaf640 10 monclient(hunting): picked mon.noname-c con 0x7f7b4814ff40 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.423+0000 7f7b4deaf640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b48149560 con 0x7f7b480a9930 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.423+0000 7f7b4deaf640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b48144580 con 0x7f7b48149dc0 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.423+0000 7f7b4deaf640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b481dbad0 con 0x7f7b4814ff40 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.423+0000 7f7b4deaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.423+0000 7f7b4deaf640 10 monclient(hunting): authenticate will time out at 1388.664596s 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.425+0000 7f7b47fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f7b4814ff40 0x7f7b4813a3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39290/0 (socket says 172.21.15.114:39290) 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.425+0000 7f7b47fff640 1 -- 172.21.15.114:0/2718171585 learned_addr learned my addr 172.21.15.114:0/2718171585 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.425+0000 7f7b2ffff640 10 client.?.objecter ms_handle_connect 0x7f7b480a9930 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.425+0000 7f7b2ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.426+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067697141 0 0) 0x7f7b48149560 con 0x7f7b480a9930 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.426+0000 7f7b2ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.426+0000 7f7b2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.426+0000 7f7b2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.426+0000 7f7b2ffff640 10 monclient(hunting): my global_id is 4560 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.426+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b34004eb0 con 0x7f7b480a9930 2024-03-29T10:48:43.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2319774785 0 0) 0x7f7b34004eb0 con 0x7f7b480a9930 2024-03-29T10:48:43.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b48149560 con 0x7f7b480a9930 2024-03-29T10:48:43.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7b38001970 con 0x7f7b480a9930 2024-03-29T10:48:43.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:43.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient(hunting): dump: 2024-03-29T10:48:43.410 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 client.?.objecter ms_handle_connect 0x7f7b48149dc0 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359518138 0 0) 0x7f7b48144580 con 0x7f7b48149dc0 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:43.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient(hunting): my global_id is 4522 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b34004eb0 con 0x7f7b48149dc0 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 792290128 0 0) 0x7f7b48149560 con 0x7f7b480a9930 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 >> v1:172.21.15.114:6790/0 conn(0x7f7b4814ff40 legacy=0x7f7b4813a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 >> v1:172.21.15.176:6789/0 conn(0x7f7b48149dc0 legacy=0x7f7b481dbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 1 monclient: found mon.a 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b481dc700 con 0x7f7b480a9930 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient: _finish_auth 0 2024-03-29T10:48:43.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.427+0000 7f7b2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b4deaf640 5 monclient: authenticate success, global_id 4560 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b4deaf640 10 monclient: _renew_subs 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b4deaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b4deaf640 1 -- 172.21.15.114:0/2718171585 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4813aad0 con 0x7f7b480a9930 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b4deaf640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b4deaf640 10 monclient: _renew_subs 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b4deaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b4deaf640 1 -- 172.21.15.114:0/2718171585 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b480ab650 con 0x7f7b480a9930 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b4deaf640 1 librados: init done 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b38001c90 con 0x7f7b480a9930 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b2ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7b38004c30 con 0x7f7b480a9930 2024-03-29T10:48:43.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b2ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.428+0000 7f7b2ffff640 10 monclient: dump: 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:43.414 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:43.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.429+0000 7f7b2dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:43.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.429+0000 7f7b45ffb640 4 set_mon_vals no callback set 2024-03-29T10:48:43.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.430+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f7b38004d90 con 0x7f7b480a9930 2024-03-29T10:48:43.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.431+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f7b38084b60 con 0x7f7b480a9930 2024-03-29T10:48:43.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.431+0000 7f7b2ffff640 10 client.4560.objecter ms_dispatch 0x7f7b48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:43.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.431+0000 7f7b2ffff640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:43.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.431+0000 7f7b2ffff640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:43.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.431+0000 7f7b2ffff640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T10:48:43.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.431+0000 7f7b2dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:43.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.431+0000 7f7b2dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:43.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.431+0000 7f7b2dffb640 1 -- 172.21.15.114:0/2718171585 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b28005080 con 0x7f7b480a9930 2024-03-29T10:48:43.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.432+0000 7f7b46ffd640 10 monclient: get_auth_request con 0x7f7b3406dd00 auth_method 0 2024-03-29T10:48:43.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.433+0000 7f7b2ffff640 10 client.4560.objecter ms_handle_connect 0x7f7b3406dd00 2024-03-29T10:48:43.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.446+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b38050440 con 0x7f7b480a9930 2024-03-29T10:48:43.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.446+0000 7f7b2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:43.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.446+0000 7f7b2ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.791+0000 7f7b2dffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:43.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.791+0000 7f7b2dffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.791+0000 7f7b2dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.791+0000 7f7b2dffb640 1 -- 172.21.15.114:0/2718171585 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b28005330 con 0x7f7b480a9930 2024-03-29T10:48:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.792+0000 7f7b2ffff640 1 -- 172.21.15.114:0/2718171585 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f7b38052370 con 0x7f7b480a9930 2024-03-29T10:48:43.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.792+0000 7f7b2ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:43.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.792+0000 7f7b2ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:43.775 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:48:43.775 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.795+0000 7f7b4deaf640 10 librados: watch_flush enter 2024-03-29T10:48:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.795+0000 7f7b4deaf640 10 librados: watch_flush exit 2024-03-29T10:48:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.795+0000 7f7b4deaf640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T10:48:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.795+0000 7f7b4deaf640 10 client.4560.objecter successfully canceled tick 2024-03-29T10:48:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.796+0000 7f7b4deaf640 1 -- 172.21.15.114:0/2718171585 >> v1:172.21.15.114:6816/775044307 conn(0x7f7b3406dd00 legacy=0x7f7b340701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.796+0000 7f7b4deaf640 10 monclient: shutdown 2024-03-29T10:48:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.796+0000 7f7b4deaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:43.778 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.796+0000 7f7b4deaf640 1 -- 172.21.15.114:0/2718171585 >> v1:172.21.15.114:6789/0 conn(0x7f7b480a9930 legacy=0x7f7b48148e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.796+0000 7f7b4deaf640 1 -- 172.21.15.114:0/2718171585 shutdown_connections 2024-03-29T10:48:43.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.796+0000 7f7b4deaf640 1 -- 172.21.15.114:0/2718171585 >> 172.21.15.114:0/2718171585 conn(0x7f7b480b35e0 msgr2=0x7f7b4813d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:43.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.797+0000 7f7b4deaf640 1 -- 172.21.15.114:0/2718171585 shutdown_connections 2024-03-29T10:48:43.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.797+0000 7f7b4deaf640 1 -- 172.21.15.114:0/2718171585 wait complete. 2024-03-29T10:48:43.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:43.797+0000 7f7b4deaf640 1 librados: shutdown 2024-03-29T10:48:43.797 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:50.604 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:50.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.768+0000 7f896dfd7640 10 monclient: get_monmap_and_config 2024-03-29T10:48:50.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.768+0000 7f896dfd7640 10 monclient: build_initial_monmap 2024-03-29T10:48:50.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.768+0000 7f896dfd7640 10 monclient: monmap: 2024-03-29T10:48:50.751 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:48:50.770163+0000 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:48:50.770163+0000 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.770+0000 7f896dfd7640 1 Processor -- start 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.770+0000 7f896dfd7640 1 -- start start 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.770+0000 7f896dfd7640 10 monclient: init 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.771+0000 7f896dfd7640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.771+0000 7f896dfd7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.771+0000 7f896dfd7640 10 monclient(hunting): picked mon.noname-b con 0x7f89681457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.771+0000 7f896dfd7640 10 monclient(hunting): picked mon.noname-c con 0x7f896813f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.771+0000 7f896dfd7640 10 monclient(hunting): picked mon.noname-a con 0x7f89680a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.771+0000 7f896dfd7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89680aaef0 con 0x7f89680a9d70 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.771+0000 7f896dfd7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8968145bd0 con 0x7f896813f650 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.771+0000 7f896dfd7640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8968145eb0 con 0x7f89681457f0 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.771+0000 7f896dfd7640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.771+0000 7f896dfd7640 10 monclient(hunting): authenticate will time out at 1396.012658s 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.772+0000 7f8966ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f896813f650 0x7f89680b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:38038/0 (socket says 172.21.15.114:38038) 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.772+0000 7f8966ffd640 1 -- 172.21.15.114:0/3629232193 learned_addr learned my addr 172.21.15.114:0/3629232193 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353680052 0 0) 0x7f89680aaef0 con 0x7f89680a9d70 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): my global_id is 4563 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8950003520 con 0x7f89680a9d70 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577488061 0 0) 0x7f8968145eb0 con 0x7f89681457f0 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): my global_id is 4646 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89680aaef0 con 0x7f89681457f0 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675863578 0 0) 0x7f8968145bd0 con 0x7f896813f650 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 10 monclient(hunting): my global_id is 4525 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.773+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8968145eb0 con 0x7f896813f650 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3298118541 0 0) 0x7f8950003520 con 0x7f89680a9d70 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 >> v1:172.21.15.114:6790/0 conn(0x7f89681457f0 legacy=0x7f896813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 >> v1:172.21.15.176:6789/0 conn(0x7f896813f650 legacy=0x7f89680b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 1 monclient: found mon.noname-a 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8968146340 con 0x7f89680a9d70 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f896dfd7640 5 monclient: authenticate success, global_id 4563 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f896dfd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8960001a10 con 0x7f89680a9d70 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 10 monclient: dump: 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:50.760 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:50.760 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:50.760 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f896dfd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:50.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8960001c30 con 0x7f89680a9d70 2024-03-29T10:48:50.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:50.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 1 -- 172.21.15.114:0/3629232193 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8960001fc0 con 0x7f89680a9d70 2024-03-29T10:48:50.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f896dfd7640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:50.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f896dfd7640 4 set_mon_vals no callback set 2024-03-29T10:48:50.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:50.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.774+0000 7f89667fc640 10 monclient: dump: 2024-03-29T10:48:50.764 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:50.764 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:50.764 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.775+0000 7f896dfd7640 10 monclient: shutdown 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.775+0000 7f896dfd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.775+0000 7f896dfd7640 1 -- 172.21.15.114:0/3629232193 >> v1:172.21.15.114:6789/0 conn(0x7f89680a9d70 legacy=0x7f896814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.775+0000 7f896dfd7640 1 -- 172.21.15.114:0/3629232193 shutdown_connections 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.775+0000 7f896dfd7640 1 -- 172.21.15.114:0/3629232193 >> 172.21.15.114:0/3629232193 conn(0x7f8968066a00 msgr2=0x7f8968066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.775+0000 7f896dfd7640 1 -- 172.21.15.114:0/3629232193 shutdown_connections 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.776+0000 7f896dfd7640 1 -- 172.21.15.114:0/3629232193 wait complete. 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.777+0000 7f896dfd7640 10 monclient: build_initial_monmap 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.777+0000 7f896dfd7640 10 monclient: monmap: 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.777+0000 7f896dfd7640 1 librados: starting msgr at 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.777+0000 7f896dfd7640 1 librados: starting objecter 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.777+0000 7f896dfd7640 1 Processor -- start 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.777+0000 7f896dfd7640 1 -- start start 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.777+0000 7f896dfd7640 1 librados: setting wanted keys 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.777+0000 7f896dfd7640 1 librados: calling monclient init 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.777+0000 7f896dfd7640 10 monclient: init 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.778+0000 7f896dfd7640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.778+0000 7f896dfd7640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.778+0000 7f896dfd7640 10 monclient(hunting): picked mon.noname-b con 0x7f89680a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f896dfd7640 10 monclient(hunting): picked mon.noname-a con 0x7f896813f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f896dfd7640 10 monclient(hunting): picked mon.noname-c con 0x7f89681457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f896dfd7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8968142b50 con 0x7f896813f650 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f896dfd7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896806dfc0 con 0x7f89680a9d70 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f896dfd7640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89681df540 con 0x7f89681457f0 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f896dfd7640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f89677fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f89680a9d70 0x7f89681dee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:39456/0 (socket says 172.21.15.114:39456) 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f89677fe640 1 -- 172.21.15.114:0/955819527 learned_addr learned my addr 172.21.15.114:0/955819527 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f896dfd7640 10 monclient(hunting): authenticate will time out at 1396.020278s 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 10 client.?.objecter ms_handle_connect 0x7f89680a9d70 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950236951 0 0) 0x7f896806dfc0 con 0x7f89680a9d70 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 10 monclient(hunting): my global_id is 4528 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8954004eb0 con 0x7f89680a9d70 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 10 client.?.objecter ms_handle_connect 0x7f89681457f0 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 10 client.?.objecter ms_handle_connect 0x7f896813f650 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.779+0000 7f894ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294935645 0 0) 0x7f89681df540 con 0x7f89681457f0 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 10 monclient(hunting): my global_id is 4649 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f896806dfc0 con 0x7f89681457f0 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 342713993 0 0) 0x7f8954004eb0 con 0x7f89680a9d70 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89681df540 con 0x7f89680a9d70 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8958001910 con 0x7f89680a9d70 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 10 monclient(hunting): dump: 2024-03-29T10:48:50.769 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.780+0000 7f894ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561707723 0 0) 0x7f8968142b50 con 0x7f896813f650 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 10 monclient(hunting): my global_id is 4566 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8954004eb0 con 0x7f896813f650 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1274029422 0 0) 0x7f896806dfc0 con 0x7f89681457f0 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8968142b50 con 0x7f89681457f0 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8960001950 con 0x7f89681457f0 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 10 monclient(hunting): dump: 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:50.771 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3540291811 0 0) 0x7f89681df540 con 0x7f89680a9d70 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 >> v1:172.21.15.114:6790/0 conn(0x7f89681457f0 legacy=0x7f896813ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 >> v1:172.21.15.114:6789/0 conn(0x7f896813f650 legacy=0x7f896813a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 1 monclient: found mon.b 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89681df6d0 con 0x7f89680a9d70 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 10 monclient: _finish_auth 0 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:50.772 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f894ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:50.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f896dfd7640 5 monclient: authenticate success, global_id 4528 2024-03-29T10:48:50.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f896dfd7640 10 monclient: _renew_subs 2024-03-29T10:48:50.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f896dfd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:50.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f896dfd7640 1 -- 172.21.15.114:0/955819527 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89681e7110 con 0x7f89680a9d70 2024-03-29T10:48:50.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f896dfd7640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:50.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f896dfd7640 10 monclient: _renew_subs 2024-03-29T10:48:50.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f896dfd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:50.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f896dfd7640 1 -- 172.21.15.114:0/955819527 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89681e7480 con 0x7f89680a9d70 2024-03-29T10:48:50.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.781+0000 7f896dfd7640 1 librados: init done 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.782+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89580035f0 con 0x7f89680a9d70 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.782+0000 7f894ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.782+0000 7f8965ffb640 4 set_mon_vals no callback set 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.783+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f8958003980 con 0x7f89680a9d70 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.783+0000 7f894ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.783+0000 7f894ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.783+0000 7f896dfd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.783+0000 7f894ffff640 10 monclient: dump: 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:50.775 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.784+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f8958003ba0 con 0x7f89680a9d70 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.784+0000 7f896dfd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.785+0000 7f896dfd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.785+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f8958084980 con 0x7f89680a9d70 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.785+0000 7f894ffff640 10 client.4528.objecter ms_dispatch 0x7f8968000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.785+0000 7f894ffff640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.785+0000 7f894ffff640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.786+0000 7f894ffff640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.786+0000 7f896dfd7640 1 -- 172.21.15.114:0/955819527 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8948005080 con 0x7f89680a9d70 2024-03-29T10:48:50.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.786+0000 7f8966ffd640 10 monclient: get_auth_request con 0x7f895406e070 auth_method 0 2024-03-29T10:48:50.777 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.786+0000 7f894ffff640 10 client.4528.objecter ms_handle_connect 0x7f895406e070 2024-03-29T10:48:50.777 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.793+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8958050260 con 0x7f89680a9d70 2024-03-29T10:48:50.777 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.793+0000 7f894ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:50.777 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:50.793+0000 7f894ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:51.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.122+0000 7f896dfd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:51.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.122+0000 7f896dfd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:51.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.122+0000 7f896dfd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:48:51.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.123+0000 7f896dfd7640 1 -- 172.21.15.114:0/955819527 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8948005330 con 0x7f89680a9d70 2024-03-29T10:48:51.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.123+0000 7f894ffff640 1 -- 172.21.15.114:0/955819527 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f8958052190 con 0x7f89680a9d70 2024-03-29T10:48:51.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.124+0000 7f894ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:51.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.124+0000 7f894ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:51.106 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:48:51.106 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:51.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.127+0000 7f896dfd7640 10 librados: watch_flush enter 2024-03-29T10:48:51.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.127+0000 7f896dfd7640 10 librados: watch_flush exit 2024-03-29T10:48:51.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.127+0000 7f896dfd7640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T10:48:51.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.127+0000 7f896dfd7640 10 client.4528.objecter successfully canceled tick 2024-03-29T10:48:51.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.127+0000 7f896dfd7640 1 -- 172.21.15.114:0/955819527 >> v1:172.21.15.114:6816/775044307 conn(0x7f895406e070 legacy=0x7f8954070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:51.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.127+0000 7f896dfd7640 10 monclient: shutdown 2024-03-29T10:48:51.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.127+0000 7f896dfd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:51.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.127+0000 7f896dfd7640 1 -- 172.21.15.114:0/955819527 >> v1:172.21.15.176:6789/0 conn(0x7f89680a9d70 legacy=0x7f89681dee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:51.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.129+0000 7f896dfd7640 1 -- 172.21.15.114:0/955819527 shutdown_connections 2024-03-29T10:48:51.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.129+0000 7f896dfd7640 1 -- 172.21.15.114:0/955819527 >> 172.21.15.114:0/955819527 conn(0x7f8968066a00 msgr2=0x7f89681d2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:51.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.130+0000 7f896dfd7640 1 -- 172.21.15.114:0/955819527 shutdown_connections 2024-03-29T10:48:51.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.130+0000 7f896dfd7640 1 -- 172.21.15.114:0/955819527 wait complete. 2024-03-29T10:48:51.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:51.130+0000 7f896dfd7640 1 librados: shutdown 2024-03-29T10:48:51.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:57.937 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:58.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.106+0000 7f71b7c63640 10 monclient: get_monmap_and_config 2024-03-29T10:48:58.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.106+0000 7f71b7c63640 10 monclient: build_initial_monmap 2024-03-29T10:48:58.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.107+0000 7f71b7c63640 10 monclient: monmap: 2024-03-29T10:48:58.089 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:58.090 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:58.090 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:48:58.108261+0000 2024-03-29T10:48:58.090 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:48:58.108261+0000 2024-03-29T10:48:58.090 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:58.090 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:58.090 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:58.090 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:48:58.090 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:48:58.090 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:58.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.108+0000 7f71b7c63640 1 Processor -- start 2024-03-29T10:48:58.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.108+0000 7f71b7c63640 1 -- start start 2024-03-29T10:48:58.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.108+0000 7f71b7c63640 10 monclient: init 2024-03-29T10:48:58.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b7c63640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:58.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b7c63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:58.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b7c63640 10 monclient(hunting): picked mon.noname-b con 0x7f71b0071de0 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:58.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b7c63640 10 monclient(hunting): picked mon.noname-c con 0x7f71b00721c0 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:58.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b7c63640 10 monclient(hunting): picked mon.noname-a con 0x7f71b0145ae0 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:58.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b7c63640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b0145650 con 0x7f71b0145ae0 2024-03-29T10:48:58.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b7c63640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b00ae260 con 0x7f71b00721c0 2024-03-29T10:48:58.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b6200640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f71b0145ae0 0x7f71b0145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54622/0 (socket says 172.21.15.114:54622) 2024-03-29T10:48:58.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b6200640 1 -- 172.21.15.114:0/3751499265 learned_addr learned my addr 172.21.15.114:0/3751499265 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:58.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b7c63640 1 -- 172.21.15.114:0/3751499265 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b01420d0 con 0x7f71b0071de0 2024-03-29T10:48:58.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b7c63640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:58.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.109+0000 7f71b7c63640 10 monclient(hunting): authenticate will time out at 1403.350877s 2024-03-29T10:48:58.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470883358 0 0) 0x7f71b0145650 con 0x7f71b0145ae0 2024-03-29T10:48:58.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:58.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): my global_id is 4569 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f719c003520 con 0x7f71b0145ae0 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537191533 0 0) 0x7f71b01420d0 con 0x7f71b0071de0 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): my global_id is 4652 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b0145650 con 0x7f71b0071de0 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3087886758 0 0) 0x7f719c003520 con 0x7f71b0145ae0 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 >> v1:172.21.15.114:6790/0 conn(0x7f71b0071de0 legacy=0x7f71b00aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 >> v1:172.21.15.176:6789/0 conn(0x7f71b00721c0 legacy=0x7f71b013f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:58.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 monclient: found mon.noname-a 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b00b0590 con 0x7f71b0145ae0 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient: _finish_auth 0 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f71a8001a10 con 0x7f71b0145ae0 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient: dump: 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:58.096 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b7c63640 5 monclient: authenticate success, global_id 4569 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b7c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b6200640 1 -- 172.21.15.114:0/3751499265 >> v1:172.21.15.114:6789/0 conn(0x7f71b0145ae0 legacy=0x7f71b0145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b6200640 1 -- 172.21.15.114:0/3751499265 >> v1:172.21.15.114:6789/0 conn(0x7f71b0145ae0 legacy=0x7f71b0145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b6200640 1 --1- 172.21.15.114:0/3751499265 >> v1:172.21.15.114:6789/0 conn(0x7f71b0145ae0 0x7f71b0145f50 :-1 s=OPENED pgs=985024 cs=1 l=1).handle_message read tag failed 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b6200640 1 --1- 172.21.15.114:0/3751499265 >> v1:172.21.15.114:6789/0 conn(0x7f71b0145ae0 0x7f71b0145f50 :-1 s=OPENED pgs=985024 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient: ms_handle_reset current mon v1:172.21.15.114:6789/0 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:58.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 >> v1:172.21.15.114:6789/0 conn(0x7f71b0145ae0 legacy=0x7f71b0145f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): picked mon.a con 0x7f719c003c80 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): picked mon.b con 0x7f719c007a10 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): picked mon.c con 0x7f71b0071de0 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f719c003520 con 0x7f719c003c80 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f71b00ae260 con 0x7f719c007a10 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f71b01420d0 con 0x7f71b0071de0 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.110+0000 7f71b49fd640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.111+0000 7f71b49fd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.114:6789/0 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.111+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428170065 0 0) 0x7f71b01420d0 con 0x7f71b0071de0 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.111+0000 7f71b49fd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.111+0000 7f71b49fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:58.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.111+0000 7f71b49fd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:48:58.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.111+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 --> v1:172.21.15.114:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f719c00f120 con 0x7f71b0071de0 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.111+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643016094 0 0) 0x7f719c003520 con 0x7f719c003c80 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 --> v1:172.21.15.114:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f71b01420d0 con 0x7f719c003c80 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3374498060 0 0) 0x7f719c00f120 con 0x7f71b0071de0 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 >> v1:172.21.15.176:6789/0 conn(0x7f719c007a10 legacy=0x7f719c009e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 >> v1:172.21.15.114:6789/0 conn(0x7f719c003c80 legacy=0x7f719c004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 1 monclient: found mon.c 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f719c0123b0 con 0x7f71b0071de0 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 10 monclient: _finish_auth 0 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f71a8004c20 con 0x7f71b0071de0 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 10 monclient: dump: 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:58.105 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:58.105 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:58.105 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:58.105 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:58.105 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:58.105 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:58.105 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:58.105 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:58.105 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b7c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 1 -- 172.21.15.114:0/3751499265 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71a8004d80 con 0x7f71b0071de0 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.112+0000 7f71b49fd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.113+0000 7f71b7c63640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.113+0000 7f71b7c63640 4 set_mon_vals no callback set 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.113+0000 7f71b7c63640 10 monclient: shutdown 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.113+0000 7f71b7c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.113+0000 7f71b7c63640 1 -- 172.21.15.114:0/3751499265 >> v1:172.21.15.114:6790/0 conn(0x7f71b0071de0 legacy=0x7f719c00d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.114+0000 7f71b7c63640 1 -- 172.21.15.114:0/3751499265 shutdown_connections 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.114+0000 7f71b7c63640 1 -- 172.21.15.114:0/3751499265 >> 172.21.15.114:0/3751499265 conn(0x7f71b006b660 msgr2=0x7f71b006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.114+0000 7f71b7c63640 1 -- 172.21.15.114:0/3751499265 shutdown_connections 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.114+0000 7f71b7c63640 1 -- 172.21.15.114:0/3751499265 wait complete. 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.115+0000 7f71b7c63640 10 monclient: build_initial_monmap 2024-03-29T10:48:58.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.115+0000 7f71b7c63640 10 monclient: monmap: 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.115+0000 7f71b7c63640 1 librados: starting msgr at 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.115+0000 7f71b7c63640 1 librados: starting objecter 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.115+0000 7f71b7c63640 1 Processor -- start 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 1 -- start start 2024-03-29T10:48:58.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 1 librados: setting wanted keys 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 1 librados: calling monclient init 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 10 monclient: init 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 10 monclient(hunting): picked mon.noname-b con 0x7f719c003c80 addr v1:172.21.15.176:6789/0 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 10 monclient(hunting): picked mon.noname-a con 0x7f719c007a10 addr v1:172.21.15.114:6789/0 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 10 monclient(hunting): picked mon.noname-c con 0x7f71b0071de0 addr v1:172.21.15.114:6790/0 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b011c970 con 0x7f719c007a10 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b0121740 con 0x7f719c003c80 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b6200640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f71b0071de0 0x7f71b01d6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:38916/0 (socket says 172.21.15.114:38916) 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b6200640 1 -- 172.21.15.114:0/1241115357 learned_addr learned my addr 172.21.15.114:0/1241115357 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 1 -- 172.21.15.114:0/1241115357 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b0120940 con 0x7f71b0071de0 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:58.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71b7c63640 10 monclient(hunting): authenticate will time out at 1403.357716s 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71ae7fc640 10 client.?.objecter ms_handle_connect 0x7f71b0071de0 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.116+0000 7f71ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.117+0000 7f71ae7fc640 10 client.?.objecter ms_handle_connect 0x7f719c007a10 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.117+0000 7f71ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.117+0000 7f71ae7fc640 10 client.?.objecter ms_handle_connect 0x7f719c003c80 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.117+0000 7f71ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.117+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266413479 0 0) 0x7f71b011c970 con 0x7f719c007a10 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.117+0000 7f71ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.117+0000 7f71ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.117+0000 7f71ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.117+0000 7f71ae7fc640 10 monclient(hunting): my global_id is 4572 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.117+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7190004eb0 con 0x7f719c007a10 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.118+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 870219749 0 0) 0x7f7190004eb0 con 0x7f719c007a10 2024-03-29T10:48:58.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.118+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71b011c970 con 0x7f719c007a10 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.118+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f71a00017f0 con 0x7f719c007a10 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.118+0000 7f71ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.118+0000 7f71ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.118+0000 7f71ae7fc640 10 monclient(hunting): dump: 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:58.110 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.118+0000 7f71ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182922952 0 0) 0x7f71b0120940 con 0x7f71b0071de0 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 10 monclient(hunting): my global_id is 4655 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7190004eb0 con 0x7f71b0071de0 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3800961681 0 0) 0x7f71b011c970 con 0x7f719c007a10 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 >> v1:172.21.15.114:6790/0 conn(0x7f71b0071de0 legacy=0x7f71b01d6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 >> v1:172.21.15.176:6789/0 conn(0x7f719c003c80 legacy=0x7f71b013e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 1 monclient: found mon.a 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:58.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b013f160 con 0x7f719c007a10 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71b7c63640 5 monclient: authenticate success, global_id 4572 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71b7c63640 10 monclient: _renew_subs 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71b7c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71b7c63640 1 -- 172.21.15.114:0/1241115357 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b011f350 con 0x7f719c007a10 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71b7c63640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71b7c63640 10 monclient: _renew_subs 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71b7c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71b7c63640 1 -- 172.21.15.114:0/1241115357 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b013ef70 con 0x7f719c007a10 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.119+0000 7f71b7c63640 1 librados: init done 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.120+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71a0001d40 con 0x7f719c007a10 2024-03-29T10:48:58.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.120+0000 7f71ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.120+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f71a0003790 con 0x7f719c007a10 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.120+0000 7f71ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.120+0000 7f71ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.120+0000 7f71ae7fc640 10 monclient: dump: 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.120+0000 7f71affff640 4 set_mon_vals no callback set 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.121+0000 7f71a7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.121+0000 7f71a7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.121+0000 7f71a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.121+0000 7f71a7fff640 1 -- 172.21.15.114:0/1241115357 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7188005080 con 0x7f719c007a10 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.122+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f71a0019760 con 0x7f719c007a10 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.123+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f71a0083740 con 0x7f719c007a10 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.123+0000 7f71ae7fc640 10 client.4572.objecter ms_dispatch 0x7f71b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.123+0000 7f71ae7fc640 3 client.4572.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.123+0000 7f71ae7fc640 3 client.4572.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.123+0000 7f71ae7fc640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.124+0000 7f71b59ff640 10 monclient: get_auth_request con 0x7f719006db10 auth_method 0 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.124+0000 7f71ae7fc640 10 client.4572.objecter ms_handle_connect 0x7f719006db10 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.131+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f71a004f060 con 0x7f719c007a10 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.131+0000 7f71ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.131+0000 7f71ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:58.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.502+0000 7f71a7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:58.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.502+0000 7f71a7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:58.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.502+0000 7f71a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:48:58.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.502+0000 7f71a7fff640 1 -- 172.21.15.114:0/1241115357 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7188005330 con 0x7f719c007a10 2024-03-29T10:48:58.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.503+0000 7f71ae7fc640 1 -- 172.21.15.114:0/1241115357 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f71a0050f90 con 0x7f719c007a10 2024-03-29T10:48:58.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.503+0000 7f71ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:58.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.503+0000 7f71ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:58.486 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:48:58.487 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:58.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 10 librados: watch_flush enter 2024-03-29T10:48:58.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 10 librados: watch_flush exit 2024-03-29T10:48:58.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-29T10:48:58.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 10 client.4572.objecter successfully canceled tick 2024-03-29T10:48:58.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 1 -- 172.21.15.114:0/1241115357 >> v1:172.21.15.114:6816/775044307 conn(0x7f719006db10 legacy=0x7f719006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 10 monclient: shutdown 2024-03-29T10:48:58.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:58.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 1 -- 172.21.15.114:0/1241115357 >> v1:172.21.15.114:6789/0 conn(0x7f719c007a10 legacy=0x7f71b011fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 1 -- 172.21.15.114:0/1241115357 shutdown_connections 2024-03-29T10:48:58.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 1 -- 172.21.15.114:0/1241115357 >> 172.21.15.114:0/1241115357 conn(0x7f71b006b660 msgr2=0x7f71b00b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:58.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 1 -- 172.21.15.114:0/1241115357 shutdown_connections 2024-03-29T10:48:58.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 1 -- 172.21.15.114:0/1241115357 wait complete. 2024-03-29T10:48:58.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:48:58.507+0000 7f71a7fff640 1 librados: shutdown 2024-03-29T10:48:58.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:05.312 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:05.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.474+0000 7fb469ec8640 10 monclient: get_monmap_and_config 2024-03-29T10:49:05.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.474+0000 7fb469ec8640 10 monclient: build_initial_monmap 2024-03-29T10:49:05.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.474+0000 7fb469ec8640 10 monclient: monmap: 2024-03-29T10:49:05.456 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:05.456 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:05.456 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:49:05.475031+0000 2024-03-29T10:49:05.456 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:49:05.475031+0000 2024-03-29T10:49:05.456 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:05.456 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:05.456 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:05.456 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:49:05.457 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:49:05.457 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:05.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.476+0000 7fb469ec8640 1 Processor -- start 2024-03-29T10:49:05.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.476+0000 7fb469ec8640 1 -- start start 2024-03-29T10:49:05.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.476+0000 7fb469ec8640 10 monclient: init 2024-03-29T10:49:05.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.477+0000 7fb469ec8640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:05.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.477+0000 7fb469ec8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:05.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.477+0000 7fb469ec8640 10 monclient(hunting): picked mon.noname-c con 0x7fb4641457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:05.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.477+0000 7fb469ec8640 10 monclient(hunting): picked mon.noname-a con 0x7fb46413f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:05.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.477+0000 7fb469ec8640 10 monclient(hunting): picked mon.noname-b con 0x7fb4640a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:05.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.477+0000 7fb469ec8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4640aaef0 con 0x7fb46413f650 2024-03-29T10:49:05.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.477+0000 7fb469ec8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb464145bd0 con 0x7fb4641457f0 2024-03-29T10:49:05.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.477+0000 7fb469ec8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb464145eb0 con 0x7fb4640a9d70 2024-03-29T10:49:05.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.477+0000 7fb469ec8640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:05.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.477+0000 7fb469ec8640 10 monclient(hunting): authenticate will time out at 1410.717899s 2024-03-29T10:49:05.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.478+0000 7fb463fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fb4640a9d70 0x7fb46414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55980/0 (socket says 172.21.15.114:55980) 2024-03-29T10:49:05.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.478+0000 7fb463fff640 1 -- 172.21.15.114:0/729981804 learned_addr learned my addr 172.21.15.114:0/729981804 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:05.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.478+0000 7fb4627fc640 1 -- 172.21.15.114:0/729981804 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262494422 0 0) 0x7fb464145eb0 con 0x7fb4640a9d70 2024-03-29T10:49:05.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.478+0000 7fb4627fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.478+0000 7fb4627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.478+0000 7fb4627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.478+0000 7fb4627fc640 10 monclient(hunting): my global_id is 4658 2024-03-29T10:49:05.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.478+0000 7fb4627fc640 1 -- 172.21.15.114:0/729981804 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb44c003520 con 0x7fb4640a9d70 2024-03-29T10:49:05.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.479+0000 7fb4627fc640 1 -- 172.21.15.114:0/729981804 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1221106804 0 0) 0x7fb44c003520 con 0x7fb4640a9d70 2024-03-29T10:49:05.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.479+0000 7fb4627fc640 1 -- 172.21.15.114:0/729981804 >> v1:172.21.15.176:6789/0 conn(0x7fb4641457f0 legacy=0x7fb46413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:05.462 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.481+0000 7fb4627fc640 1 -- 172.21.15.114:0/729981804 >> v1:172.21.15.114:6789/0 conn(0x7fb46413f650 legacy=0x7fb4640b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:05.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.481+0000 7fb4627fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:05.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.481+0000 7fb4627fc640 1 monclient: found mon.noname-b 2024-03-29T10:49:05.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.481+0000 7fb4627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:05.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.481+0000 7fb4627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:49:05.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.481+0000 7fb4627fc640 1 -- 172.21.15.114:0/729981804 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb464146340 con 0x7fb4640a9d70 2024-03-29T10:49:05.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 1 -- 172.21.15.114:0/729981804 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb458001690 con 0x7fb4640a9d70 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 10 monclient: dump: 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:05.465 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 1 -- 172.21.15.114:0/729981804 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb458001920 con 0x7fb4640a9d70 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 1 -- 172.21.15.114:0/729981804 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb458001cb0 con 0x7fb4640a9d70 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb4627fc640 10 monclient: dump: 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.466 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:05.467 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:05.467 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:05.467 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:05.467 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:05.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb469ec8640 5 monclient: authenticate success, global_id 4658 2024-03-29T10:49:05.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.482+0000 7fb469ec8640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:05.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.483+0000 7fb469ec8640 4 set_mon_vals no callback set 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.484+0000 7fb469ec8640 10 monclient: shutdown 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.484+0000 7fb469ec8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.484+0000 7fb469ec8640 1 -- 172.21.15.114:0/729981804 >> v1:172.21.15.114:6790/0 conn(0x7fb4640a9d70 legacy=0x7fb46414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.485+0000 7fb469ec8640 1 -- 172.21.15.114:0/729981804 shutdown_connections 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.485+0000 7fb469ec8640 1 -- 172.21.15.114:0/729981804 >> 172.21.15.114:0/729981804 conn(0x7fb464066a00 msgr2=0x7fb464066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.485+0000 7fb469ec8640 1 -- 172.21.15.114:0/729981804 shutdown_connections 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.485+0000 7fb469ec8640 1 -- 172.21.15.114:0/729981804 wait complete. 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.486+0000 7fb469ec8640 10 monclient: build_initial_monmap 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.486+0000 7fb469ec8640 10 monclient: monmap: 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:05.469 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.487+0000 7fb469ec8640 1 librados: starting msgr at 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.487+0000 7fb469ec8640 1 librados: starting objecter 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.487+0000 7fb469ec8640 1 Processor -- start 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.487+0000 7fb469ec8640 1 -- start start 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.487+0000 7fb469ec8640 1 librados: setting wanted keys 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.488+0000 7fb469ec8640 1 librados: calling monclient init 2024-03-29T10:49:05.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.488+0000 7fb469ec8640 10 monclient: init 2024-03-29T10:49:05.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.489+0000 7fb469ec8640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:05.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.489+0000 7fb469ec8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:05.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.489+0000 7fb469ec8640 10 monclient(hunting): picked mon.noname-c con 0x7fb4640a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:05.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.489+0000 7fb469ec8640 10 monclient(hunting): picked mon.noname-b con 0x7fb46413f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:05.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.489+0000 7fb469ec8640 10 monclient(hunting): picked mon.noname-a con 0x7fb4641457f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:05.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.489+0000 7fb469ec8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46413e3a0 con 0x7fb4641457f0 2024-03-29T10:49:05.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.489+0000 7fb469ec8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46413e120 con 0x7fb46413f650 2024-03-29T10:49:05.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.489+0000 7fb469ec8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46413cd20 con 0x7fb4640a9d70 2024-03-29T10:49:05.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.489+0000 7fb469ec8640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:05.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.489+0000 7fb469ec8640 10 monclient(hunting): authenticate will time out at 1410.729597s 2024-03-29T10:49:05.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.492+0000 7fb4637fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fb4640a9d70 0x7fb46413c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34752/0 (socket says 172.21.15.114:34752) 2024-03-29T10:49:05.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.492+0000 7fb4637fe640 1 -- 172.21.15.114:0/4120364730 learned_addr learned my addr 172.21.15.114:0/4120364730 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:05.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 client.?.objecter ms_handle_connect 0x7fb4640a9d70 2024-03-29T10:49:05.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:05.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 client.?.objecter ms_handle_connect 0x7fb46413f650 2024-03-29T10:49:05.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:05.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309728625 0 0) 0x7fb46413cd20 con 0x7fb4640a9d70 2024-03-29T10:49:05.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): my global_id is 4661 2024-03-29T10:49:05.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb450004eb0 con 0x7fb4640a9d70 2024-03-29T10:49:05.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252202251 0 0) 0x7fb46413e120 con 0x7fb46413f650 2024-03-29T10:49:05.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): my global_id is 4531 2024-03-29T10:49:05.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb46413cd20 con 0x7fb46413f650 2024-03-29T10:49:05.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 452177410 0 0) 0x7fb450004eb0 con 0x7fb4640a9d70 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb450005490 con 0x7fb4640a9d70 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb45c001b50 con 0x7fb4640a9d70 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): dump: 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:05.482 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.493+0000 7fb44bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 10 client.?.objecter ms_handle_connect 0x7fb4641457f0 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2468744250 0 0) 0x7fb450005490 con 0x7fb4640a9d70 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 >> v1:172.21.15.176:6789/0 conn(0x7fb46413f650 legacy=0x7fb4641dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 >> v1:172.21.15.114:6789/0 conn(0x7fb4641457f0 legacy=0x7fb4641e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 1 monclient: found mon.c 2024-03-29T10:49:05.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:05.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:05.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb464141450 con 0x7fb4640a9d70 2024-03-29T10:49:05.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 10 monclient: _finish_auth 0 2024-03-29T10:49:05.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:05.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:05.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb45c002060 con 0x7fb4640a9d70 2024-03-29T10:49:05.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:05.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fb45c001b50 con 0x7fb4640a9d70 2024-03-29T10:49:05.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:05.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb44bfff640 10 monclient: dump: 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.494+0000 7fb461ffb640 4 set_mon_vals no callback set 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.495+0000 7fb469ec8640 5 monclient: authenticate success, global_id 4661 2024-03-29T10:49:05.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.495+0000 7fb469ec8640 10 monclient: _renew_subs 2024-03-29T10:49:05.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.495+0000 7fb469ec8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:05.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.495+0000 7fb469ec8640 1 -- 172.21.15.114:0/4120364730 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb464141730 con 0x7fb4640a9d70 2024-03-29T10:49:05.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.496+0000 7fb469ec8640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:05.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.496+0000 7fb469ec8640 10 monclient: _renew_subs 2024-03-29T10:49:05.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.496+0000 7fb469ec8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:05.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.496+0000 7fb469ec8640 1 -- 172.21.15.114:0/4120364730 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb464142b60 con 0x7fb4640a9d70 2024-03-29T10:49:05.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.496+0000 7fb469ec8640 1 librados: init done 2024-03-29T10:49:05.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.498+0000 7fb469ec8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:05.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.498+0000 7fb469ec8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:05.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.498+0000 7fb469ec8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:05.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.498+0000 7fb469ec8640 1 -- 172.21.15.114:0/4120364730 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb464142e60 con 0x7fb4640a9d70 2024-03-29T10:49:05.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.500+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fb45c01aad0 con 0x7fb4640a9d70 2024-03-29T10:49:05.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.501+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7fb45c0847e0 con 0x7fb4640a9d70 2024-03-29T10:49:05.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.501+0000 7fb44bfff640 10 client.4661.objecter ms_dispatch 0x7fb464000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:05.487 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.501+0000 7fb44bfff640 3 client.4661.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:05.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.501+0000 7fb44bfff640 3 client.4661.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:05.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.501+0000 7fb44bfff640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T10:49:05.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.501+0000 7fb462ffd640 10 monclient: get_auth_request con 0x7fb45006e2e0 auth_method 0 2024-03-29T10:49:05.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.502+0000 7fb44bfff640 10 client.4661.objecter ms_handle_connect 0x7fb45006e2e0 2024-03-29T10:49:05.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.508+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb45c050030 con 0x7fb4640a9d70 2024-03-29T10:49:05.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.508+0000 7fb44bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:05.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.508+0000 7fb44bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:05.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.822+0000 7fb469ec8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:05.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.822+0000 7fb469ec8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:05.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.822+0000 7fb469ec8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:05.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.822+0000 7fb469ec8640 1 -- 172.21.15.114:0/4120364730 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb464075f10 con 0x7fb4640a9d70 2024-03-29T10:49:05.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.823+0000 7fb44bfff640 1 -- 172.21.15.114:0/4120364730 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7fb45c052130 con 0x7fb4640a9d70 2024-03-29T10:49:05.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.823+0000 7fb44bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:05.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.823+0000 7fb44bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:05.806 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:49:05.806 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:05.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.826+0000 7fb469ec8640 10 librados: watch_flush enter 2024-03-29T10:49:05.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.827+0000 7fb469ec8640 10 librados: watch_flush exit 2024-03-29T10:49:05.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.827+0000 7fb469ec8640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T10:49:05.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.827+0000 7fb469ec8640 10 client.4661.objecter successfully canceled tick 2024-03-29T10:49:05.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.827+0000 7fb469ec8640 1 -- 172.21.15.114:0/4120364730 >> v1:172.21.15.114:6816/775044307 conn(0x7fb45006e2e0 legacy=0x7fb4500707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.827+0000 7fb469ec8640 10 monclient: shutdown 2024-03-29T10:49:05.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.827+0000 7fb469ec8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:05.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.827+0000 7fb469ec8640 1 -- 172.21.15.114:0/4120364730 >> v1:172.21.15.114:6790/0 conn(0x7fb4640a9d70 legacy=0x7fb46413c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.829+0000 7fb469ec8640 1 -- 172.21.15.114:0/4120364730 shutdown_connections 2024-03-29T10:49:05.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.829+0000 7fb469ec8640 1 -- 172.21.15.114:0/4120364730 >> 172.21.15.114:0/4120364730 conn(0x7fb464066a00 msgr2=0x7fb4640b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:05.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.829+0000 7fb469ec8640 1 -- 172.21.15.114:0/4120364730 shutdown_connections 2024-03-29T10:49:05.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.829+0000 7fb469ec8640 1 -- 172.21.15.114:0/4120364730 wait complete. 2024-03-29T10:49:05.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:05.830+0000 7fb469ec8640 1 librados: shutdown 2024-03-29T10:49:05.828 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:12.636 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:12.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.804+0000 7f2ac8ba8640 10 monclient: get_monmap_and_config 2024-03-29T10:49:12.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.804+0000 7f2ac8ba8640 10 monclient: build_initial_monmap 2024-03-29T10:49:12.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.804+0000 7f2ac8ba8640 10 monclient: monmap: 2024-03-29T10:49:12.786 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:12.786 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:12.786 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:49:12.804942+0000 2024-03-29T10:49:12.786 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:49:12.804942+0000 2024-03-29T10:49:12.786 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:12.786 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:12.787 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:12.787 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:49:12.787 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:49:12.787 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:12.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.805+0000 7f2ac8ba8640 1 Processor -- start 2024-03-29T10:49:12.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.805+0000 7f2ac8ba8640 1 -- start start 2024-03-29T10:49:12.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.805+0000 7f2ac8ba8640 10 monclient: init 2024-03-29T10:49:12.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.806+0000 7f2ac8ba8640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:12.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.806+0000 7f2ac8ba8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:12.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.810+0000 7f2ac8ba8640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac40b0770 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:12.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.810+0000 7f2ac8ba8640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac414a700 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:12.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.811+0000 7f2ac8ba8640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac40ac5a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:12.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.811+0000 7f2ac8ba8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac40aba70 con 0x7f2ac40ac5a0 2024-03-29T10:49:12.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.811+0000 7f2ac8ba8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac40aecb0 con 0x7f2ac40b0770 2024-03-29T10:49:12.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.811+0000 7f2ac8ba8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac40af150 con 0x7f2ac414a700 2024-03-29T10:49:12.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.811+0000 7f2ac8ba8640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:12.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.811+0000 7f2ac8ba8640 10 monclient(hunting): authenticate will time out at 1418.051053s 2024-03-29T10:49:12.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.813+0000 7f2ac1d9b640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f2ac414a700 0x7f2ac40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:46409/0 (socket says 172.21.15.114:46409) 2024-03-29T10:49:12.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.813+0000 7f2ac1d9b640 1 -- 172.21.15.114:0/3233140051 learned_addr learned my addr 172.21.15.114:0/3233140051 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:12.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.814+0000 7f2ac159a640 1 -- 172.21.15.114:0/3233140051 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921930186 0 0) 0x7f2ac40af150 con 0x7f2ac414a700 2024-03-29T10:49:12.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.814+0000 7f2ac159a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:12.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.814+0000 7f2ac159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:12.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.814+0000 7f2ac159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:12.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.814+0000 7f2ac159a640 10 monclient(hunting): my global_id is 4664 2024-03-29T10:49:12.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.814+0000 7f2ac159a640 1 -- 172.21.15.114:0/3233140051 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aac003520 con 0x7f2ac414a700 2024-03-29T10:49:12.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 1 -- 172.21.15.114:0/3233140051 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3023291644 0 0) 0x7f2aac003520 con 0x7f2ac414a700 2024-03-29T10:49:12.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 1 -- 172.21.15.114:0/3233140051 >> v1:172.21.15.176:6789/0 conn(0x7f2ac40b0770 legacy=0x7f2ac4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 1 -- 172.21.15.114:0/3233140051 >> v1:172.21.15.114:6789/0 conn(0x7f2ac40ac5a0 legacy=0x7f2ac4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:12.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 1 monclient: found mon.noname-b 2024-03-29T10:49:12.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 1 -- 172.21.15.114:0/3233140051 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac40ac9d0 con 0x7f2ac414a700 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 10 monclient: _finish_auth 0 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 1 -- 172.21.15.114:0/3233140051 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2ab8001690 con 0x7f2ac414a700 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 10 monclient: dump: 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:12.798 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac8ba8640 5 monclient: authenticate success, global_id 4664 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac8ba8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 1 -- 172.21.15.114:0/3233140051 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ab80017f0 con 0x7f2ac414a700 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 1 -- 172.21.15.114:0/3233140051 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2ab8001c40 con 0x7f2ac414a700 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.815+0000 7f2ac159a640 10 monclient: dump: 2024-03-29T10:49:12.799 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:12.800 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:12.800 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:12.800 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:12.800 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.800 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:12.800 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:12.800 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:12.800 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:12.800 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.816+0000 7f2ac8ba8640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.816+0000 7f2ac8ba8640 4 set_mon_vals no callback set 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.816+0000 7f2ac8ba8640 10 monclient: shutdown 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.816+0000 7f2ac8ba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.816+0000 7f2ac8ba8640 1 -- 172.21.15.114:0/3233140051 >> v1:172.21.15.114:6790/0 conn(0x7f2ac414a700 legacy=0x7f2ac40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.817+0000 7f2ac8ba8640 1 -- 172.21.15.114:0/3233140051 shutdown_connections 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.817+0000 7f2ac8ba8640 1 -- 172.21.15.114:0/3233140051 >> 172.21.15.114:0/3233140051 conn(0x7f2ac406b660 msgr2=0x7f2ac406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.817+0000 7f2ac8ba8640 1 -- 172.21.15.114:0/3233140051 shutdown_connections 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.817+0000 7f2ac8ba8640 1 -- 172.21.15.114:0/3233140051 wait complete. 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.818+0000 7f2ac8ba8640 10 monclient: build_initial_monmap 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.818+0000 7f2ac8ba8640 10 monclient: monmap: 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:12.803 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.819+0000 7f2ac8ba8640 1 librados: starting msgr at 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.819+0000 7f2ac8ba8640 1 librados: starting objecter 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.819+0000 7f2ac8ba8640 1 Processor -- start 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.819+0000 7f2ac8ba8640 1 -- start start 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.819+0000 7f2ac8ba8640 1 librados: setting wanted keys 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.819+0000 7f2ac8ba8640 1 librados: calling monclient init 2024-03-29T10:49:12.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.819+0000 7f2ac8ba8640 10 monclient: init 2024-03-29T10:49:12.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.820+0000 7f2ac8ba8640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:12.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.820+0000 7f2ac8ba8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:12.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.820+0000 7f2ac8ba8640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac40ac5a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:12.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.820+0000 7f2ac8ba8640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac40b0770 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:12.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.820+0000 7f2ac8ba8640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac414a700 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:12.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.820+0000 7f2ac8ba8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac411fd40 con 0x7f2ac40ac5a0 2024-03-29T10:49:12.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.820+0000 7f2ac8ba8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac41e7740 con 0x7f2ac414a700 2024-03-29T10:49:12.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.820+0000 7f2ac8ba8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac411ece0 con 0x7f2ac40b0770 2024-03-29T10:49:12.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.820+0000 7f2ac8ba8640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:12.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.820+0000 7f2ac8ba8640 10 monclient(hunting): authenticate will time out at 1418.060129s 2024-03-29T10:49:12.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.824+0000 7f2ac259c640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2ac40ac5a0 0x7f2ac411e5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46315/0 (socket says 172.21.15.114:46315) 2024-03-29T10:49:12.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.824+0000 7f2ac259c640 1 -- 172.21.15.114:0/1868863322 learned_addr learned my addr 172.21.15.114:0/1868863322 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:12.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 10 client.?.objecter ms_handle_connect 0x7f2ac40ac5a0 2024-03-29T10:49:12.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:12.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 10 client.?.objecter ms_handle_connect 0x7f2ac40b0770 2024-03-29T10:49:12.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:12.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 10 client.?.objecter ms_handle_connect 0x7f2ac414a700 2024-03-29T10:49:12.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:12.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161153754 0 0) 0x7f2ac411fd40 con 0x7f2ac40ac5a0 2024-03-29T10:49:12.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:12.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:12.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:12.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 10 monclient(hunting): my global_id is 4578 2024-03-29T10:49:12.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.828+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa4004eb0 con 0x7f2ac40ac5a0 2024-03-29T10:49:12.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036979372 0 0) 0x7f2ac411ece0 con 0x7f2ac40b0770 2024-03-29T10:49:12.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:12.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:12.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:12.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 10 monclient(hunting): my global_id is 4667 2024-03-29T10:49:12.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac411fd40 con 0x7f2ac40b0770 2024-03-29T10:49:12.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1405181070 0 0) 0x7f2ac411fd40 con 0x7f2ac40b0770 2024-03-29T10:49:12.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ac411ece0 con 0x7f2ac40b0770 2024-03-29T10:49:12.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2ab8001b00 con 0x7f2ac40b0770 2024-03-29T10:49:12.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:12.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:12.813 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:12.813 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:12.813 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:12.813 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.829+0000 7f2ab2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.830+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 578096395 0 0) 0x7f2ac411ece0 con 0x7f2ac40b0770 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.830+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 >> v1:172.21.15.176:6789/0 conn(0x7f2ac414a700 legacy=0x7f2ac41eec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.830+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 >> v1:172.21.15.114:6789/0 conn(0x7f2ac40ac5a0 legacy=0x7f2ac411e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.830+0000 7f2ab2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.830+0000 7f2ab2ffd640 1 monclient: found mon.c 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.830+0000 7f2ab2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:12.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.830+0000 7f2ab2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:12.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.830+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac411b1c0 con 0x7f2ac40b0770 2024-03-29T10:49:12.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.830+0000 7f2ab2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:12.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.830+0000 7f2ab2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:12.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.830+0000 7f2ab2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.831+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ab8001cd0 con 0x7f2ac40b0770 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.831+0000 7f2ab2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.831+0000 7f2ac0d99640 4 set_mon_vals no callback set 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.832+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2ab8004c30 con 0x7f2ac40b0770 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.832+0000 7f2ab2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.832+0000 7f2ab2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.832+0000 7f2ab2ffd640 10 monclient: dump: 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:12.816 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:12.817 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:12.817 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:12.817 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:12.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.833+0000 7f2ac8ba8640 5 monclient: authenticate success, global_id 4667 2024-03-29T10:49:12.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.833+0000 7f2ac8ba8640 10 monclient: _renew_subs 2024-03-29T10:49:12.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.833+0000 7f2ac8ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:12.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.834+0000 7f2ac8ba8640 1 -- 172.21.15.114:0/1868863322 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac41e8370 con 0x7f2ac40b0770 2024-03-29T10:49:12.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.834+0000 7f2ac8ba8640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:12.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.834+0000 7f2ac8ba8640 10 monclient: _renew_subs 2024-03-29T10:49:12.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.834+0000 7f2ac8ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:12.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.834+0000 7f2ac8ba8640 1 -- 172.21.15.114:0/1868863322 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac411db30 con 0x7f2ac40b0770 2024-03-29T10:49:12.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.834+0000 7f2ac8ba8640 1 librados: init done 2024-03-29T10:49:12.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.835+0000 7f2ab0ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:12.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.835+0000 7f2ab0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:12.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.835+0000 7f2ab0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:12.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.835+0000 7f2ab0ff9640 1 -- 172.21.15.114:0/1868863322 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9c005080 con 0x7f2ac40b0770 2024-03-29T10:49:12.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.836+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f2ab801ac00 con 0x7f2ac40b0770 2024-03-29T10:49:12.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.837+0000 7f2ac259c640 10 monclient: get_auth_request con 0x7f2aa406dba0 auth_method 0 2024-03-29T10:49:12.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.837+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3339045211 0 0) 0x7f2ab80848b0 con 0x7f2ac40b0770 2024-03-29T10:49:12.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.837+0000 7f2ab2ffd640 10 client.4667.objecter ms_dispatch 0x7f2ac4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:12.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.837+0000 7f2ab2ffd640 3 client.4667.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:12.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.837+0000 7f2ab2ffd640 3 client.4667.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:12.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.837+0000 7f2ab2ffd640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-29T10:49:12.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.837+0000 7f2ab2ffd640 10 client.4667.objecter ms_handle_connect 0x7f2aa406dba0 2024-03-29T10:49:12.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.846+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ab8050100 con 0x7f2ac40b0770 2024-03-29T10:49:12.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.847+0000 7f2ab2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:12.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:12.847+0000 7f2ab2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:13.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.186+0000 7f2ab0ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:13.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.186+0000 7f2ab0ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:13.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.186+0000 7f2ab0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:13.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.186+0000 7f2ab0ff9640 1 -- 172.21.15.114:0/1868863322 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a9c005330 con 0x7f2ac40b0770 2024-03-29T10:49:13.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.188+0000 7f2ab2ffd640 1 -- 172.21.15.114:0/1868863322 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2884837540) 0x7f2ab8052200 con 0x7f2ac40b0770 2024-03-29T10:49:13.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.188+0000 7f2ab2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:13.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.188+0000 7f2ab2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:13.171 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:49:13.171 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:39:16.160461+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:16.160461+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:13.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.195+0000 7f2ab0ff9640 10 librados: watch_flush enter 2024-03-29T10:49:13.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.195+0000 7f2ab0ff9640 10 librados: watch_flush exit 2024-03-29T10:49:13.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.195+0000 7f2ab0ff9640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-29T10:49:13.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.195+0000 7f2ab0ff9640 10 client.4667.objecter successfully canceled tick 2024-03-29T10:49:13.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.196+0000 7f2ab0ff9640 1 -- 172.21.15.114:0/1868863322 >> v1:172.21.15.114:6816/775044307 conn(0x7f2aa406dba0 legacy=0x7f2aa4070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:13.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.196+0000 7f2ab0ff9640 10 monclient: shutdown 2024-03-29T10:49:13.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.196+0000 7f2ab0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:13.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.196+0000 7f2ab0ff9640 1 -- 172.21.15.114:0/1868863322 >> v1:172.21.15.114:6790/0 conn(0x7f2ac40b0770 legacy=0x7f2ac41e7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:13.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.196+0000 7f2ab0ff9640 1 -- 172.21.15.114:0/1868863322 shutdown_connections 2024-03-29T10:49:13.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.196+0000 7f2ab0ff9640 1 -- 172.21.15.114:0/1868863322 >> 172.21.15.114:0/1868863322 conn(0x7f2ac406b660 msgr2=0x7f2ac406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:13.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.196+0000 7f2ab0ff9640 1 -- 172.21.15.114:0/1868863322 shutdown_connections 2024-03-29T10:49:13.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.196+0000 7f2ab0ff9640 1 -- 172.21.15.114:0/1868863322 wait complete. 2024-03-29T10:49:13.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:13.198+0000 7f2ab0ff9640 1 librados: shutdown 2024-03-29T10:49:13.196 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:20.003 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:20.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.170+0000 7f24a35f6640 10 monclient: get_monmap_and_config 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.170+0000 7f24a35f6640 10 monclient: build_initial_monmap 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.171+0000 7f24a35f6640 10 monclient: monmap: 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:49:20.172269+0000 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:49:20.172269+0000 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:20.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.172+0000 7f24a35f6640 1 Processor -- start 2024-03-29T10:49:20.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.172+0000 7f24a35f6640 1 -- start start 2024-03-29T10:49:20.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.172+0000 7f24a35f6640 10 monclient: init 2024-03-29T10:49:20.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.173+0000 7f24a35f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:20.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.173+0000 7f24a35f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:20.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.173+0000 7f24a35f6640 10 monclient(hunting): picked mon.noname-c con 0x7f2494006430 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:20.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.173+0000 7f24a35f6640 10 monclient(hunting): picked mon.noname-b con 0x7f249402a660 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:20.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.173+0000 7f24a35f6640 10 monclient(hunting): picked mon.noname-a con 0x7f249402b390 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:20.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.173+0000 7f24a35f6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249402a140 con 0x7f249402b390 2024-03-29T10:49:20.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.173+0000 7f24a35f6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940c75d0 con 0x7f2494006430 2024-03-29T10:49:20.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.173+0000 7f24a35f6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2494033130 con 0x7f249402a660 2024-03-29T10:49:20.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.173+0000 7f24a35f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:20.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.173+0000 7f24a35f6640 10 monclient(hunting): authenticate will time out at 1425.414907s 2024-03-29T10:49:20.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.174+0000 7f24a0b91640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f249402a660 0x7f24940c6ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:37604/0 (socket says 172.21.15.114:37604) 2024-03-29T10:49:20.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.174+0000 7f24a0b91640 1 -- 172.21.15.114:0/962697174 learned_addr learned my addr 172.21.15.114:0/962697174 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:20.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.174+0000 7f2493fff640 1 -- 172.21.15.114:0/962697174 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576423784 0 0) 0x7f2494033130 con 0x7f249402a660 2024-03-29T10:49:20.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.174+0000 7f2493fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:20.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.174+0000 7f2493fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:20.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.174+0000 7f2493fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:20.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.174+0000 7f2493fff640 10 monclient(hunting): my global_id is 4670 2024-03-29T10:49:20.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.174+0000 7f2493fff640 1 -- 172.21.15.114:0/962697174 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2488003520 con 0x7f249402a660 2024-03-29T10:49:20.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 1 -- 172.21.15.114:0/962697174 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1351250022 0 0) 0x7f2488003520 con 0x7f249402a660 2024-03-29T10:49:20.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 1 -- 172.21.15.114:0/962697174 >> v1:172.21.15.176:6789/0 conn(0x7f2494006430 legacy=0x7f2494034600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:20.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 1 -- 172.21.15.114:0/962697174 >> v1:172.21.15.114:6789/0 conn(0x7f249402b390 legacy=0x7f24940320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:20.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 1 monclient: found mon.noname-b 2024-03-29T10:49:20.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:20.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:49:20.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 1 -- 172.21.15.114:0/962697174 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24940332c0 con 0x7f249402a660 2024-03-29T10:49:20.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 10 monclient: _finish_auth 0 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f24a35f6640 5 monclient: authenticate success, global_id 4670 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f24a35f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 1 -- 172.21.15.114:0/962697174 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f248c001690 con 0x7f249402a660 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 10 monclient: dump: 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:20.165 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f24a35f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 1 -- 172.21.15.114:0/962697174 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f248c0018b0 con 0x7f249402a660 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f2493fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f24a35f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.179+0000 7f24a35f6640 4 set_mon_vals no callback set 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.180+0000 7f2493fff640 1 -- 172.21.15.114:0/962697174 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f248c001c40 con 0x7f249402a660 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.180+0000 7f2493fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:20.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.180+0000 7f2493fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.180+0000 7f2493fff640 10 monclient: dump: 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.180+0000 7f24a35f6640 10 monclient: shutdown 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.180+0000 7f24a35f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:20.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.180+0000 7f24a35f6640 1 -- 172.21.15.114:0/962697174 >> v1:172.21.15.114:6790/0 conn(0x7f249402a660 legacy=0x7f24940c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.181+0000 7f24a35f6640 1 -- 172.21.15.114:0/962697174 shutdown_connections 2024-03-29T10:49:20.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.181+0000 7f24a35f6640 1 -- 172.21.15.114:0/962697174 >> 172.21.15.114:0/962697174 conn(0x7f2494005910 msgr2=0x7f2494005d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:20.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.183+0000 7f24a35f6640 1 -- 172.21.15.114:0/962697174 shutdown_connections 2024-03-29T10:49:20.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.186+0000 7f24a35f6640 1 -- 172.21.15.114:0/962697174 wait complete. 2024-03-29T10:49:20.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.187+0000 7f24a35f6640 10 monclient: build_initial_monmap 2024-03-29T10:49:20.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.187+0000 7f24a35f6640 10 monclient: monmap: 2024-03-29T10:49:20.172 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:20.172 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:20.172 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:49:20.172 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:49:20.172 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:20.172 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:20.172 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.187+0000 7f24a35f6640 1 librados: starting msgr at 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.187+0000 7f24a35f6640 1 librados: starting objecter 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.187+0000 7f24a35f6640 1 Processor -- start 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.188+0000 7f24a35f6640 1 -- start start 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.188+0000 7f24a35f6640 1 librados: setting wanted keys 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.188+0000 7f24a35f6640 1 librados: calling monclient init 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.188+0000 7f24a35f6640 10 monclient: init 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.188+0000 7f24a35f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.188+0000 7f24a35f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:20.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.188+0000 7f24a35f6640 10 monclient(hunting): picked mon.noname-c con 0x7f2494006430 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:20.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.190+0000 7f24a35f6640 10 monclient(hunting): picked mon.noname-b con 0x7f249402a660 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:20.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.190+0000 7f24a35f6640 10 monclient(hunting): picked mon.noname-a con 0x7f249402b390 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:20.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.190+0000 7f24a35f6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940bd8c0 con 0x7f249402b390 2024-03-29T10:49:20.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.190+0000 7f24a35f6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249402eac0 con 0x7f249402a660 2024-03-29T10:49:20.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.190+0000 7f24a35f6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249402f970 con 0x7f2494006430 2024-03-29T10:49:20.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.190+0000 7f24a35f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:20.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.190+0000 7f24a35f6640 10 monclient(hunting): authenticate will time out at 1425.431822s 2024-03-29T10:49:20.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.191+0000 7f24a1392640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f2494006430 0x7f249415fa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:43256/0 (socket says 172.21.15.114:43256) 2024-03-29T10:49:20.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.191+0000 7f24a1392640 1 -- 172.21.15.114:0/3250536009 learned_addr learned my addr 172.21.15.114:0/3250536009 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:20.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.192+0000 7f2491ffb640 10 client.?.objecter ms_handle_connect 0x7f2494006430 2024-03-29T10:49:20.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.192+0000 7f2491ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:20.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.192+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342662359 0 0) 0x7f249402f970 con 0x7f2494006430 2024-03-29T10:49:20.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.192+0000 7f2491ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:20.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.192+0000 7f2491ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:20.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.192+0000 7f2491ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:20.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.192+0000 7f2491ffb640 10 monclient(hunting): my global_id is 4673 2024-03-29T10:49:20.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.192+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2480004eb0 con 0x7f2494006430 2024-03-29T10:49:20.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 client.?.objecter ms_handle_connect 0x7f249402a660 2024-03-29T10:49:20.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:20.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 965622209 0 0) 0x7f2480004eb0 con 0x7f2494006430 2024-03-29T10:49:20.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f249402f970 con 0x7f2494006430 2024-03-29T10:49:20.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f24980017d0 con 0x7f2494006430 2024-03-29T10:49:20.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:20.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:20.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 monclient(hunting): dump: 2024-03-29T10:49:20.180 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:20.180 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845390248 0 0) 0x7f249402eac0 con 0x7f249402a660 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:20.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 monclient(hunting): my global_id is 4540 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2480004eb0 con 0x7f249402a660 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3301116703 0 0) 0x7f2480004eb0 con 0x7f249402a660 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f249402eac0 con 0x7f249402a660 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f248c0013b0 con 0x7f249402a660 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.195+0000 7f2491ffb640 10 monclient(hunting): dump: 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:20.182 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3419419350 0 0) 0x7f249402eac0 con 0x7f249402a660 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 >> v1:172.21.15.114:6790/0 conn(0x7f2494006430 legacy=0x7f249415fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 >> v1:172.21.15.114:6789/0 conn(0x7f249402b390 legacy=0x7f24940bd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 1 monclient: found mon.b 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249402fc50 con 0x7f249402a660 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 10 monclient: _finish_auth 0 2024-03-29T10:49:20.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f24a35f6640 5 monclient: authenticate success, global_id 4540 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f24a35f6640 10 monclient: _renew_subs 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f24a35f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f24a35f6640 1 -- 172.21.15.114:0/3250536009 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249402e5a0 con 0x7f249402a660 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f24a35f6640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f24a35f6640 10 monclient: _renew_subs 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f24a35f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f24a35f6640 1 -- 172.21.15.114:0/3250536009 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f249402daa0 con 0x7f249402a660 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f24a35f6640 1 librados: init done 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f248c004940 con 0x7f249402a660 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2491ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.196+0000 7f2492ffd640 4 set_mon_vals no callback set 2024-03-29T10:49:20.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.197+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f248c004c50 con 0x7f249402a660 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.197+0000 7f2491ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.197+0000 7f2491ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.197+0000 7f2491ffb640 10 monclient: dump: 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:20.185 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:20.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.198+0000 7f24a35f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:20.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.198+0000 7f24a35f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:20.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.198+0000 7f24a35f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:49:20.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.198+0000 7f24a35f6640 1 -- 172.21.15.114:0/3250536009 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247c005080 con 0x7f249402a660 2024-03-29T10:49:20.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.199+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f248c01ac20 con 0x7f249402a660 2024-03-29T10:49:20.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.200+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1764351048 0 0) 0x7f248c0848f0 con 0x7f249402a660 2024-03-29T10:49:20.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.200+0000 7f2491ffb640 10 client.4540.objecter ms_dispatch 0x7f249c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:20.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.200+0000 7f2491ffb640 3 client.4540.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:20.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.200+0000 7f2491ffb640 3 client.4540.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:20.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.200+0000 7f2491ffb640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-29T10:49:20.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.202+0000 7f24a1392640 10 monclient: get_auth_request con 0x7f248006dd90 auth_method 0 2024-03-29T10:49:20.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.203+0000 7f2491ffb640 10 client.4540.objecter ms_handle_connect 0x7f248006dd90 2024-03-29T10:49:20.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.206+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f248c0501d0 con 0x7f249402a660 2024-03-29T10:49:20.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.206+0000 7f2491ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:20.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.206+0000 7f2491ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:20.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.531+0000 7f24a35f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:20.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.531+0000 7f24a35f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:20.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.531+0000 7f24a35f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:49:20.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.531+0000 7f24a35f6640 1 -- 172.21.15.114:0/3250536009 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f247c005330 con 0x7f249402a660 2024-03-29T10:49:20.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.532+0000 7f2491ffb640 1 -- 172.21.15.114:0/3250536009 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2201894833) 0x7f248c052100 con 0x7f249402a660 2024-03-29T10:49:20.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.532+0000 7f2491ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:20.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.532+0000 7f2491ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:20.515 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:49:20.515 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:49:18.930439+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:18.930439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:20.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.535+0000 7f24a35f6640 10 librados: watch_flush enter 2024-03-29T10:49:20.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.535+0000 7f24a35f6640 10 librados: watch_flush exit 2024-03-29T10:49:20.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.535+0000 7f24a35f6640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-29T10:49:20.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.535+0000 7f24a35f6640 10 client.4540.objecter successfully canceled tick 2024-03-29T10:49:20.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.535+0000 7f24a35f6640 1 -- 172.21.15.114:0/3250536009 >> v1:172.21.15.114:6816/775044307 conn(0x7f248006dd90 legacy=0x7f2480070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.535+0000 7f24a35f6640 10 monclient: shutdown 2024-03-29T10:49:20.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.535+0000 7f24a35f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:20.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.535+0000 7f24a35f6640 1 -- 172.21.15.114:0/3250536009 >> v1:172.21.15.176:6789/0 conn(0x7f249402a660 legacy=0x7f24940bca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.536+0000 7f24a35f6640 1 -- 172.21.15.114:0/3250536009 shutdown_connections 2024-03-29T10:49:20.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.536+0000 7f24a35f6640 1 -- 172.21.15.114:0/3250536009 >> 172.21.15.114:0/3250536009 conn(0x7f2494005910 msgr2=0x7f24940043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:20.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.537+0000 7f24a35f6640 1 -- 172.21.15.114:0/3250536009 shutdown_connections 2024-03-29T10:49:20.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.537+0000 7f24a35f6640 1 -- 172.21.15.114:0/3250536009 wait complete. 2024-03-29T10:49:20.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:20.537+0000 7f24a35f6640 1 librados: shutdown 2024-03-29T10:49:20.536 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:27.344 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:27.501 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.519+0000 7f049d779640 10 monclient: get_monmap_and_config 2024-03-29T10:49:27.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.519+0000 7f049d779640 10 monclient: build_initial_monmap 2024-03-29T10:49:27.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.519+0000 7f049d779640 10 monclient: monmap: 2024-03-29T10:49:27.502 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:27.502 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:27.502 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:49:27.520855+0000 2024-03-29T10:49:27.502 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:49:27.520855+0000 2024-03-29T10:49:27.503 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:27.503 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:27.503 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:27.503 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:49:27.503 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:49:27.503 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:27.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.521+0000 7f049d779640 1 Processor -- start 2024-03-29T10:49:27.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.521+0000 7f049d779640 1 -- start start 2024-03-29T10:49:27.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.521+0000 7f049d779640 10 monclient: init 2024-03-29T10:49:27.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.522+0000 7f049d779640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:27.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.522+0000 7f049d779640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:27.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.522+0000 7f049d779640 10 monclient(hunting): picked mon.noname-a con 0x7f0490038c50 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:27.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.522+0000 7f049d779640 10 monclient(hunting): picked mon.noname-b con 0x7f04900360d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:27.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.522+0000 7f049d779640 10 monclient(hunting): picked mon.noname-c con 0x7f04900c8f80 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:27.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.522+0000 7f049d779640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0490030b10 con 0x7f0490038c50 2024-03-29T10:49:27.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.522+0000 7f049d779640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04900cc0c0 con 0x7f04900c8f80 2024-03-29T10:49:27.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.522+0000 7f049d779640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0490036540 con 0x7f04900360d0 2024-03-29T10:49:27.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.522+0000 7f049d779640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:27.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.522+0000 7f049d779640 10 monclient(hunting): authenticate will time out at 1432.763456s 2024-03-29T10:49:27.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.523+0000 7f04977fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f04900360d0 0x7f0490034090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35016/0 (socket says 172.21.15.114:35016) 2024-03-29T10:49:27.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.523+0000 7f04977fe640 1 -- 172.21.15.114:0/2966244085 learned_addr learned my addr 172.21.15.114:0/2966244085 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:27.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.523+0000 7f0496ffd640 1 -- 172.21.15.114:0/2966244085 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414375722 0 0) 0x7f0490036540 con 0x7f04900360d0 2024-03-29T10:49:27.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.523+0000 7f0496ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:27.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.523+0000 7f0496ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:27.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.523+0000 7f0496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:27.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.523+0000 7f0496ffd640 10 monclient(hunting): my global_id is 4676 2024-03-29T10:49:27.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.523+0000 7f0496ffd640 1 -- 172.21.15.114:0/2966244085 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0488003520 con 0x7f04900360d0 2024-03-29T10:49:27.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.523+0000 7f0496ffd640 1 -- 172.21.15.114:0/2966244085 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3141467538 0 0) 0x7f0488003520 con 0x7f04900360d0 2024-03-29T10:49:27.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.523+0000 7f0496ffd640 1 -- 172.21.15.114:0/2966244085 >> v1:172.21.15.176:6789/0 conn(0x7f04900c8f80 legacy=0x7f04900d7570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:27.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.523+0000 7f0496ffd640 1 -- 172.21.15.114:0/2966244085 >> v1:172.21.15.114:6789/0 conn(0x7f0490038c50 legacy=0x7f04900cc5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:27.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 1 monclient: found mon.noname-b 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 1 -- 172.21.15.114:0/2966244085 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04900cbc50 con 0x7f04900360d0 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 1 -- 172.21.15.114:0/2966244085 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0484001690 con 0x7f04900360d0 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f0496ffd640 10 monclient: dump: 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:27.508 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:27.509 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:27.509 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:27.509 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.509 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:27.509 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:27.509 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:27.509 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:27.509 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:27.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f049d779640 5 monclient: authenticate success, global_id 4676 2024-03-29T10:49:27.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.525+0000 7f049d779640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:27.509 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.527+0000 7f0496ffd640 1 -- 172.21.15.114:0/2966244085 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0484001860 con 0x7f04900360d0 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.527+0000 7f0496ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.527+0000 7f0496ffd640 1 -- 172.21.15.114:0/2966244085 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f0484001cb0 con 0x7f04900360d0 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.527+0000 7f0496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.527+0000 7f0496ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.527+0000 7f0496ffd640 10 monclient: dump: 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:27.510 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:27.511 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:27.511 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:27.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.527+0000 7f049d779640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:27.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.527+0000 7f049d779640 4 set_mon_vals no callback set 2024-03-29T10:49:27.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.527+0000 7f049d779640 10 monclient: shutdown 2024-03-29T10:49:27.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.528+0000 7f049d779640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:27.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.528+0000 7f049d779640 1 -- 172.21.15.114:0/2966244085 >> v1:172.21.15.114:6790/0 conn(0x7f04900360d0 legacy=0x7f0490034090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.528+0000 7f049d779640 1 -- 172.21.15.114:0/2966244085 shutdown_connections 2024-03-29T10:49:27.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.528+0000 7f049d779640 1 -- 172.21.15.114:0/2966244085 >> 172.21.15.114:0/2966244085 conn(0x7f0490005c00 msgr2=0x7f0490006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:27.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.529+0000 7f049d779640 1 -- 172.21.15.114:0/2966244085 shutdown_connections 2024-03-29T10:49:27.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.531+0000 7f049d779640 1 -- 172.21.15.114:0/2966244085 wait complete. 2024-03-29T10:49:27.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.531+0000 7f049d779640 10 monclient: build_initial_monmap 2024-03-29T10:49:27.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.531+0000 7f049d779640 10 monclient: monmap: 2024-03-29T10:49:27.515 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:27.515 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:27.515 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:49:27.515 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:49:27.515 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:27.516 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:27.516 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:27.516 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:49:27.516 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:49:27.516 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:27.516 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.532+0000 7f049d779640 1 librados: starting msgr at 2024-03-29T10:49:27.516 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.532+0000 7f049d779640 1 librados: starting objecter 2024-03-29T10:49:27.516 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.532+0000 7f049d779640 1 Processor -- start 2024-03-29T10:49:27.517 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.532+0000 7f049d779640 1 -- start start 2024-03-29T10:49:27.517 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.532+0000 7f049d779640 1 librados: setting wanted keys 2024-03-29T10:49:27.517 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.532+0000 7f049d779640 1 librados: calling monclient init 2024-03-29T10:49:27.517 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.532+0000 7f049d779640 10 monclient: init 2024-03-29T10:49:27.517 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.533+0000 7f049d779640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.533+0000 7f049d779640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.533+0000 7f049d779640 10 monclient(hunting): picked mon.noname-c con 0x7f04900360d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.533+0000 7f049d779640 10 monclient(hunting): picked mon.noname-b con 0x7f0490038c50 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.533+0000 7f049d779640 10 monclient(hunting): picked mon.noname-a con 0x7f04900c8f80 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.533+0000 7f049d779640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0490160c00 con 0x7f04900c8f80 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.533+0000 7f049d779640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0490032010 con 0x7f0490038c50 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.533+0000 7f049d779640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04900321a0 con 0x7f04900360d0 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.533+0000 7f049d779640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.533+0000 7f049d779640 10 monclient(hunting): authenticate will time out at 1432.774616s 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.534+0000 7f0497fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f04900360d0 0x7f04900ce6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42720/0 (socket says 172.21.15.114:42720) 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.534+0000 7f0497fff640 1 -- 172.21.15.114:0/1096706910 learned_addr learned my addr 172.21.15.114:0/1096706910 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.534+0000 7f0494ff9640 10 client.?.objecter ms_handle_connect 0x7f04900360d0 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.534+0000 7f0494ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:27.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.534+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886184486 0 0) 0x7f04900321a0 con 0x7f04900360d0 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.534+0000 7f0494ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.534+0000 7f0494ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.534+0000 7f0494ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.534+0000 7f0494ff9640 10 monclient(hunting): my global_id is 4679 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.534+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f047c004eb0 con 0x7f04900360d0 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.535+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2192694227 0 0) 0x7f047c004eb0 con 0x7f04900360d0 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.535+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04900321a0 con 0x7f04900360d0 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.535+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f048c001960 con 0x7f04900360d0 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.535+0000 7f0494ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.535+0000 7f0494ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.535+0000 7f0494ff9640 10 monclient(hunting): dump: 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:27.519 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:27.520 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:27.520 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.520 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:27.520 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:27.520 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:27.520 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:27.520 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:27.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.535+0000 7f0494ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:27.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.535+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3901960480 0 0) 0x7f04900321a0 con 0x7f04900360d0 2024-03-29T10:49:27.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.535+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 >> v1:172.21.15.176:6789/0 conn(0x7f0490038c50 legacy=0x7f04901610e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:27.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.535+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 >> v1:172.21.15.114:6789/0 conn(0x7f04900c8f80 legacy=0x7f049016b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:27.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.536+0000 7f0494ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:49:27.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.536+0000 7f0494ff9640 1 monclient: found mon.c 2024-03-29T10:49:27.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.536+0000 7f0494ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:27.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.536+0000 7f0494ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:27.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.536+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04900cedb0 con 0x7f04900360d0 2024-03-29T10:49:27.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.536+0000 7f0494ff9640 10 monclient: _finish_auth 0 2024-03-29T10:49:27.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.536+0000 7f0494ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:49:27.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.536+0000 7f0494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:27.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.537+0000 7f049d779640 5 monclient: authenticate success, global_id 4679 2024-03-29T10:49:27.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.537+0000 7f049d779640 10 monclient: _renew_subs 2024-03-29T10:49:27.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.537+0000 7f049d779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.537+0000 7f049d779640 1 -- 172.21.15.114:0/1096706910 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04901617f0 con 0x7f04900360d0 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.537+0000 7f049d779640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.537+0000 7f049d779640 10 monclient: _renew_subs 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.537+0000 7f049d779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.537+0000 7f049d779640 1 -- 172.21.15.114:0/1096706910 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04900cddd0 con 0x7f04900360d0 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.537+0000 7f049d779640 1 librados: init done 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.538+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f048c001c80 con 0x7f04900360d0 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.538+0000 7f0494ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.538+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f048c003a40 con 0x7f04900360d0 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.538+0000 7f0494ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.538+0000 7f0494ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.538+0000 7f0494ff9640 10 monclient: dump: 2024-03-29T10:49:27.525 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.538+0000 7f04967fc640 4 set_mon_vals no callback set 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.540+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f048c003ba0 con 0x7f04900360d0 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.540+0000 7f049d779640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.540+0000 7f049d779640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:27.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.540+0000 7f049d779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:27.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.540+0000 7f049d779640 1 -- 172.21.15.114:0/1096706910 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0478005080 con 0x7f04900360d0 2024-03-29T10:49:27.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.541+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1764351048 0 0) 0x7f048c084af0 con 0x7f04900360d0 2024-03-29T10:49:27.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.541+0000 7f0494ff9640 10 client.4679.objecter ms_dispatch 0x7f0498000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:27.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.541+0000 7f0494ff9640 3 client.4679.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:27.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.541+0000 7f0494ff9640 3 client.4679.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:27.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.541+0000 7f0494ff9640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-29T10:49:27.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.541+0000 7f04977fe640 10 monclient: get_auth_request con 0x7f047c06dcc0 auth_method 0 2024-03-29T10:49:27.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.541+0000 7f0494ff9640 10 client.4679.objecter ms_handle_connect 0x7f047c06dcc0 2024-03-29T10:49:27.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.548+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f048c0503d0 con 0x7f04900360d0 2024-03-29T10:49:27.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.548+0000 7f0494ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:27.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.548+0000 7f0494ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:27.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.878+0000 7f049d779640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:27.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.878+0000 7f049d779640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:27.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.878+0000 7f049d779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:27.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.878+0000 7f049d779640 1 -- 172.21.15.114:0/1096706910 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0478005330 con 0x7f04900360d0 2024-03-29T10:49:27.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.880+0000 7f0494ff9640 1 -- 172.21.15.114:0/1096706910 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2201894833) 0x7f048c052300 con 0x7f04900360d0 2024-03-29T10:49:27.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.880+0000 7f0494ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:27.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.880+0000 7f0494ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:27.863 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:49:27.863 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:49:18.930439+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:18.930439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:27.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.884+0000 7f049d779640 10 librados: watch_flush enter 2024-03-29T10:49:27.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.884+0000 7f049d779640 10 librados: watch_flush exit 2024-03-29T10:49:27.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.884+0000 7f049d779640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-29T10:49:27.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.884+0000 7f049d779640 10 client.4679.objecter successfully canceled tick 2024-03-29T10:49:27.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.884+0000 7f049d779640 1 -- 172.21.15.114:0/1096706910 >> v1:172.21.15.114:6816/775044307 conn(0x7f047c06dcc0 legacy=0x7f047c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.884+0000 7f049d779640 10 monclient: shutdown 2024-03-29T10:49:27.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.884+0000 7f049d779640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:27.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.884+0000 7f049d779640 1 -- 172.21.15.114:0/1096706910 >> v1:172.21.15.114:6790/0 conn(0x7f04900360d0 legacy=0x7f04900ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.886+0000 7f049d779640 1 -- 172.21.15.114:0/1096706910 shutdown_connections 2024-03-29T10:49:27.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.886+0000 7f049d779640 1 -- 172.21.15.114:0/1096706910 >> 172.21.15.114:0/1096706910 conn(0x7f0490005c00 msgr2=0x7f0490004890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:27.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.886+0000 7f049d779640 1 -- 172.21.15.114:0/1096706910 shutdown_connections 2024-03-29T10:49:27.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.887+0000 7f049d779640 1 -- 172.21.15.114:0/1096706910 wait complete. 2024-03-29T10:49:27.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:27.887+0000 7f049d779640 1 librados: shutdown 2024-03-29T10:49:27.887 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:34.694 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:34.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.860+0000 7f983c52b640 10 monclient: get_monmap_and_config 2024-03-29T10:49:34.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.860+0000 7f983c52b640 10 monclient: build_initial_monmap 2024-03-29T10:49:34.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.860+0000 7f983c52b640 10 monclient: monmap: 2024-03-29T10:49:34.843 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:34.843 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:34.843 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:49:34.861506+0000 2024-03-29T10:49:34.843 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:49:34.861506+0000 2024-03-29T10:49:34.843 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:34.843 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:34.843 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:34.843 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:49:34.844 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:49:34.844 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:34.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.860+0000 7f983c52b640 1 Processor -- start 2024-03-29T10:49:34.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.860+0000 7f983c52b640 1 -- start start 2024-03-29T10:49:34.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.860+0000 7f983c52b640 10 monclient: init 2024-03-29T10:49:34.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983c52b640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983c52b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983c52b640 10 monclient(hunting): picked mon.noname-a con 0x7f98341457f0 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983c52b640 10 monclient(hunting): picked mon.noname-b con 0x7f983413f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983c52b640 10 monclient(hunting): picked mon.noname-c con 0x7f98340a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983c52b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98340aaef0 con 0x7f98341457f0 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983c52b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9834145bd0 con 0x7f98340a9d70 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983c52b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9834145eb0 con 0x7f983413f650 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983c52b640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983c52b640 10 monclient(hunting): authenticate will time out at 1440.103157s 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983a2c7640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f98341457f0 0x7f983413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41804/0 (socket says 172.21.15.114:41804) 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f983a2c7640 1 -- 172.21.15.114:0/3132410182 learned_addr learned my addr 172.21.15.114:0/3132410182 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:34.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f98392c5640 1 -- 172.21.15.114:0/3132410182 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99749151 0 0) 0x7f98340aaef0 con 0x7f98341457f0 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f98392c5640 10 monclient(hunting): _negotiate 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f98392c5640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f98392c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f98392c5640 10 monclient(hunting): my global_id is 4584 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.862+0000 7f98392c5640 1 -- 172.21.15.114:0/3132410182 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9820003520 con 0x7f98341457f0 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 1 -- 172.21.15.114:0/3132410182 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254449215 0 0) 0x7f9834145eb0 con 0x7f983413f650 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 10 monclient(hunting): _negotiate 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 10 monclient(hunting): my global_id is 4682 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 1 -- 172.21.15.114:0/3132410182 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98200048a0 con 0x7f983413f650 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 1 -- 172.21.15.114:0/3132410182 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 877631586 0 0) 0x7f98200048a0 con 0x7f983413f650 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 1 -- 172.21.15.114:0/3132410182 >> v1:172.21.15.176:6789/0 conn(0x7f98340a9d70 legacy=0x7f983414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 1 -- 172.21.15.114:0/3132410182 >> v1:172.21.15.114:6789/0 conn(0x7f98341457f0 legacy=0x7f983413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:34.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 10 monclient: _finish_hunting 0 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 1 monclient: found mon.noname-b 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 1 -- 172.21.15.114:0/3132410182 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9834146340 con 0x7f983413f650 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 10 monclient: _finish_auth 0 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 10 monclient: _check_auth_tickets 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.863+0000 7f98392c5640 1 -- 172.21.15.114:0/3132410182 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9828001630 con 0x7f983413f650 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f98392c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f98392c5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f98392c5640 10 monclient: dump: 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:34.865 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f983c52b640 5 monclient: authenticate success, global_id 4682 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f983c52b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f98392c5640 1 -- 172.21.15.114:0/3132410182 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9828001880 con 0x7f983413f650 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f98392c5640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:34.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f983c52b640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f983c52b640 4 set_mon_vals no callback set 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f98392c5640 1 -- 172.21.15.114:0/3132410182 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9828001630 con 0x7f983413f650 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f98392c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f98392c5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.864+0000 7f98392c5640 10 monclient: dump: 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:34.867 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.865+0000 7f983c52b640 10 monclient: shutdown 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.865+0000 7f983c52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.865+0000 7f983c52b640 1 -- 172.21.15.114:0/3132410182 >> v1:172.21.15.114:6790/0 conn(0x7f983413f650 legacy=0x7f98340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.865+0000 7f983c52b640 1 -- 172.21.15.114:0/3132410182 shutdown_connections 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.865+0000 7f983c52b640 1 -- 172.21.15.114:0/3132410182 >> 172.21.15.114:0/3132410182 conn(0x7f9834066a00 msgr2=0x7f9834066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.865+0000 7f983c52b640 1 -- 172.21.15.114:0/3132410182 shutdown_connections 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.865+0000 7f983c52b640 1 -- 172.21.15.114:0/3132410182 wait complete. 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.866+0000 7f983c52b640 10 monclient: build_initial_monmap 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.866+0000 7f983c52b640 10 monclient: monmap: 2024-03-29T10:49:34.868 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:34.869 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:34.869 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:49:34.869 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:49:34.869 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:34.869 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:34.869 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:34.869 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:49:34.869 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:49:34.869 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:34.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.867+0000 7f983c52b640 1 librados: starting msgr at 2024-03-29T10:49:34.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.867+0000 7f983c52b640 1 librados: starting objecter 2024-03-29T10:49:34.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.867+0000 7f983c52b640 1 Processor -- start 2024-03-29T10:49:34.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.867+0000 7f983c52b640 1 -- start start 2024-03-29T10:49:34.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.867+0000 7f983c52b640 1 librados: setting wanted keys 2024-03-29T10:49:34.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.867+0000 7f983c52b640 1 librados: calling monclient init 2024-03-29T10:49:34.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.867+0000 7f983c52b640 10 monclient: init 2024-03-29T10:49:34.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983c52b640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:34.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983c52b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:34.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983c52b640 10 monclient(hunting): picked mon.noname-a con 0x7f98340a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:34.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983c52b640 10 monclient(hunting): picked mon.noname-c con 0x7f983413f650 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:34.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983a2c7640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f98340a9d70 0x7f98341cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42472/0 (socket says 172.21.15.114:42472) 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983a2c7640 1 -- 172.21.15.114:0/1133603823 learned_addr learned my addr 172.21.15.114:0/1133603823 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983c52b640 10 monclient(hunting): picked mon.noname-b con 0x7f98341457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f9826ffd640 10 client.?.objecter ms_handle_connect 0x7f98340a9d70 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f9826ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98341cbe60 con 0x7f98340a9d70 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98341ccd20 con 0x7f98341457f0 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98341cd000 con 0x7f983413f650 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983c52b640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.868+0000 7f983c52b640 10 monclient(hunting): authenticate will time out at 1440.109761s 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 client.?.objecter ms_handle_connect 0x7f983413f650 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 client.?.objecter ms_handle_connect 0x7f98341457f0 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:34.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721695691 0 0) 0x7f98341cbe60 con 0x7f98340a9d70 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): my global_id is 4587 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9818004eb0 con 0x7f98340a9d70 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856709085 0 0) 0x7f98341cd000 con 0x7f983413f650 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): my global_id is 4685 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98341cbe60 con 0x7f983413f650 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581990117 0 0) 0x7f98341ccd20 con 0x7f98341457f0 2024-03-29T10:49:34.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:34.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:34.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:34.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): my global_id is 4546 2024-03-29T10:49:34.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98341cd000 con 0x7f98341457f0 2024-03-29T10:49:34.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 169863417 0 0) 0x7f98341cbe60 con 0x7f983413f650 2024-03-29T10:49:34.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98341ccd20 con 0x7f983413f650 2024-03-29T10:49:34.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f9828001d30 con 0x7f983413f650 2024-03-29T10:49:34.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:34.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:34.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:34.873 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2382671654 0 0) 0x7f98341cd000 con 0x7f98341457f0 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98341cbe60 con 0x7f98341457f0 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f982c001af0 con 0x7f98341457f0 2024-03-29T10:49:34.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.869+0000 7f9826ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1777596656 0 0) 0x7f9818004eb0 con 0x7f98340a9d70 2024-03-29T10:49:34.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98341cd000 con 0x7f98340a9d70 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f98300018d0 con 0x7f98340a9d70 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:34.876 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4149103892 0 0) 0x7f98341cbe60 con 0x7f98341457f0 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 >> v1:172.21.15.114:6790/0 conn(0x7f983413f650 legacy=0x7f98341410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 >> v1:172.21.15.114:6789/0 conn(0x7f98340a9d70 legacy=0x7f98341cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 1 monclient: found mon.b 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98341cd2e0 con 0x7f98341457f0 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f9826ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f983c52b640 5 monclient: authenticate success, global_id 4546 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f983c52b640 10 monclient: _renew_subs 2024-03-29T10:49:34.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f983c52b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:49:34.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98341cc0b0 con 0x7f98341457f0 2024-03-29T10:49:34.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f983c52b640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:34.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f983c52b640 10 monclient: _renew_subs 2024-03-29T10:49:34.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f983c52b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:49:34.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98341cde10 con 0x7f98341457f0 2024-03-29T10:49:34.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.870+0000 7f983c52b640 1 librados: init done 2024-03-29T10:49:34.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.871+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f982c002000 con 0x7f98341457f0 2024-03-29T10:49:34.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.871+0000 7f9826ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:34.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.871+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f982c003b10 con 0x7f98341457f0 2024-03-29T10:49:34.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.871+0000 7f9826ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:34.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.871+0000 7f9826ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.871+0000 7f9826ffd640 10 monclient: dump: 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:34.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.871+0000 7f9838ac4640 4 set_mon_vals no callback set 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.871+0000 7f983c52b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.872+0000 7f983c52b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.872+0000 7f983c52b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.872+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98341ce0c0 con 0x7f98341457f0 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.873+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f982c019ae0 con 0x7f98341457f0 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.874+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1764351048 0 0) 0x7f982c083a90 con 0x7f98341457f0 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.874+0000 7f9826ffd640 10 client.4546.objecter ms_dispatch 0x7f9834000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.874+0000 7f9826ffd640 3 client.4546.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.874+0000 7f9826ffd640 3 client.4546.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.874+0000 7f9826ffd640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.874+0000 7f983a2c7640 10 monclient: get_auth_request con 0x7f981806e110 auth_method 0 2024-03-29T10:49:34.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.875+0000 7f9826ffd640 10 client.4546.objecter ms_handle_connect 0x7f981806e110 2024-03-29T10:49:34.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.882+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f982c04f1a0 con 0x7f98341457f0 2024-03-29T10:49:34.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.882+0000 7f9826ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:34.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:34.882+0000 7f9826ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:35.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.234+0000 7f983c52b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:35.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.234+0000 7f983c52b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:35.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.234+0000 7f983c52b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T10:49:35.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.234+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9834075f10 con 0x7f98341457f0 2024-03-29T10:49:35.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.236+0000 7f9826ffd640 1 -- 172.21.15.114:0/1133603823 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2201894833) 0x7f982c0512a0 con 0x7f98341457f0 2024-03-29T10:49:35.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.236+0000 7f9826ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:35.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.236+0000 7f9826ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:35.220 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:49:35.220 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:49:18.930439+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:18.930439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:35.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.241+0000 7f983c52b640 10 librados: watch_flush enter 2024-03-29T10:49:35.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.241+0000 7f983c52b640 10 librados: watch_flush exit 2024-03-29T10:49:35.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.241+0000 7f983c52b640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T10:49:35.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.241+0000 7f983c52b640 10 client.4546.objecter successfully canceled tick 2024-03-29T10:49:35.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.241+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 >> v1:172.21.15.114:6816/775044307 conn(0x7f981806e110 legacy=0x7f98180705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:35.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.241+0000 7f983c52b640 10 monclient: shutdown 2024-03-29T10:49:35.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.241+0000 7f983c52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:35.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.241+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 >> v1:172.21.15.176:6789/0 conn(0x7f98341457f0 legacy=0x7f98341e66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:35.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.241+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 shutdown_connections 2024-03-29T10:49:35.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.241+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 >> 172.21.15.114:0/1133603823 conn(0x7f9834066a00 msgr2=0x7f98340b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:35.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.242+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 shutdown_connections 2024-03-29T10:49:35.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.242+0000 7f983c52b640 1 -- 172.21.15.114:0/1133603823 wait complete. 2024-03-29T10:49:35.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:35.242+0000 7f983c52b640 1 librados: shutdown 2024-03-29T10:49:35.239 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:42.047 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:42.201 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.217+0000 7f39015ad640 10 monclient: get_monmap_and_config 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.217+0000 7f39015ad640 10 monclient: build_initial_monmap 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.217+0000 7f39015ad640 10 monclient: monmap: 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:49:42.218963+0000 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:49:42.218963+0000 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:42.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.218+0000 7f39015ad640 1 Processor -- start 2024-03-29T10:49:42.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.219+0000 7f39015ad640 1 -- start start 2024-03-29T10:49:42.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.219+0000 7f39015ad640 10 monclient: init 2024-03-29T10:49:42.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.220+0000 7f39015ad640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.220+0000 7f39015ad640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.220+0000 7f39015ad640 10 monclient(hunting): picked mon.noname-b con 0x7f38f4029eb0 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.220+0000 7f39015ad640 10 monclient(hunting): picked mon.noname-a con 0x7f38f402b0a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.220+0000 7f39015ad640 10 monclient(hunting): picked mon.noname-c con 0x7f38f40c9570 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.220+0000 7f39015ad640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f402a290 con 0x7f38f402b0a0 2024-03-29T10:49:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.220+0000 7f39015ad640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f40331c0 con 0x7f38f40c9570 2024-03-29T10:49:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.220+0000 7f39015ad640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f402c100 con 0x7f38f4029eb0 2024-03-29T10:49:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.220+0000 7f39015ad640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:42.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.220+0000 7f39015ad640 10 monclient(hunting): authenticate will time out at 1447.461373s 2024-03-29T10:49:42.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.224+0000 7f38fbfff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f38f4029eb0 0x7f38f40cf120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:57169/0 (socket says 172.21.15.114:57169) 2024-03-29T10:49:42.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.224+0000 7f38fbfff640 1 -- 172.21.15.114:0/1575353117 learned_addr learned my addr 172.21.15.114:0/1575353117 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:42.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.224+0000 7f38faffd640 1 -- 172.21.15.114:0/1575353117 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009717374 0 0) 0x7f38f402c100 con 0x7f38f4029eb0 2024-03-29T10:49:42.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.224+0000 7f38faffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:42.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.224+0000 7f38faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:42.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.224+0000 7f38faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:42.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.224+0000 7f38faffd640 10 monclient(hunting): my global_id is 4688 2024-03-29T10:49:42.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.225+0000 7f38faffd640 1 -- 172.21.15.114:0/1575353117 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38ec003520 con 0x7f38f4029eb0 2024-03-29T10:49:42.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.227+0000 7f38faffd640 1 -- 172.21.15.114:0/1575353117 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383404476 0 0) 0x7f38f402a290 con 0x7f38f402b0a0 2024-03-29T10:49:42.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.227+0000 7f38faffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:42.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.227+0000 7f38faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:42.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.227+0000 7f38faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:42.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.227+0000 7f38faffd640 10 monclient(hunting): my global_id is 4590 2024-03-29T10:49:42.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.227+0000 7f38faffd640 1 -- 172.21.15.114:0/1575353117 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f402c100 con 0x7f38f402b0a0 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 1 -- 172.21.15.114:0/1575353117 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1728740043 0 0) 0x7f38ec003520 con 0x7f38f4029eb0 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 1 -- 172.21.15.114:0/1575353117 >> v1:172.21.15.176:6789/0 conn(0x7f38f40c9570 legacy=0x7f38f40d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 1 -- 172.21.15.114:0/1575353117 >> v1:172.21.15.114:6789/0 conn(0x7f38f402b0a0 legacy=0x7f38f40c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 1 monclient: found mon.noname-b 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 1 -- 172.21.15.114:0/1575353117 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f402a440 con 0x7f38f4029eb0 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 1 -- 172.21.15.114:0/1575353117 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f38fc04f730 con 0x7f38f4029eb0 2024-03-29T10:49:42.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 10 monclient: dump: 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 1 -- 172.21.15.114:0/1575353117 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38fc0008d0 con 0x7f38f4029eb0 2024-03-29T10:49:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 1 -- 172.21.15.114:0/1575353117 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f38fc0758b0 con 0x7f38f4029eb0 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f38faffd640 10 monclient: dump: 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:42.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f39015ad640 5 monclient: authenticate success, global_id 4688 2024-03-29T10:49:42.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f39015ad640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:42.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.228+0000 7f39015ad640 4 set_mon_vals no callback set 2024-03-29T10:49:42.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.229+0000 7f39015ad640 10 monclient: shutdown 2024-03-29T10:49:42.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.229+0000 7f39015ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:42.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.229+0000 7f39015ad640 1 -- 172.21.15.114:0/1575353117 >> v1:172.21.15.114:6790/0 conn(0x7f38f4029eb0 legacy=0x7f38f40cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.230+0000 7f39015ad640 1 -- 172.21.15.114:0/1575353117 shutdown_connections 2024-03-29T10:49:42.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.230+0000 7f39015ad640 1 -- 172.21.15.114:0/1575353117 >> 172.21.15.114:0/1575353117 conn(0x7f38f40058f0 msgr2=0x7f38f4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:42.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.230+0000 7f39015ad640 1 -- 172.21.15.114:0/1575353117 shutdown_connections 2024-03-29T10:49:42.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.230+0000 7f39015ad640 1 -- 172.21.15.114:0/1575353117 wait complete. 2024-03-29T10:49:42.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.231+0000 7f39015ad640 10 monclient: build_initial_monmap 2024-03-29T10:49:42.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.231+0000 7f39015ad640 10 monclient: monmap: 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.231+0000 7f39015ad640 1 librados: starting msgr at 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.231+0000 7f39015ad640 1 librados: starting objecter 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.231+0000 7f39015ad640 1 Processor -- start 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 1 -- start start 2024-03-29T10:49:42.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 1 librados: setting wanted keys 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 1 librados: calling monclient init 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 10 monclient: init 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 10 monclient(hunting): picked mon.noname-c con 0x7f38f4029eb0 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 10 monclient(hunting): picked mon.noname-a con 0x7f38f402b0a0 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 10 monclient(hunting): picked mon.noname-b con 0x7f38f40c9570 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f40cbca0 con 0x7f38f402b0a0 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f402c890 con 0x7f38f40c9570 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f402ca20 con 0x7f38f4029eb0 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:42.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.232+0000 7f39015ad640 10 monclient(hunting): authenticate will time out at 1447.473933s 2024-03-29T10:49:42.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.237+0000 7f38fb7fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f38f402b0a0 0x7f38f40cc130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:57639/0 (socket says 172.21.15.114:57639) 2024-03-29T10:49:42.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.237+0000 7f38fbfff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f38f4029eb0 0x7f38f40c45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:57229/0 (socket says 172.21.15.114:57229) 2024-03-29T10:49:42.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.237+0000 7f38fbfff640 1 -- 172.21.15.114:0/445935815 learned_addr learned my addr 172.21.15.114:0/445935815 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:42.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.237+0000 7f38f8ff9640 10 client.?.objecter ms_handle_connect 0x7f38f402b0a0 2024-03-29T10:49:42.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.237+0000 7f38f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:42.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.238+0000 7f38f8ff9640 10 client.?.objecter ms_handle_connect 0x7f38f4029eb0 2024-03-29T10:49:42.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.238+0000 7f38f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:42.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.240+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906657017 0 0) 0x7f38f40cbca0 con 0x7f38f402b0a0 2024-03-29T10:49:42.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.240+0000 7f38f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:42.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.240+0000 7f38f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:42.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.240+0000 7f38f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:42.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.241+0000 7f38f8ff9640 10 monclient(hunting): my global_id is 4593 2024-03-29T10:49:42.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.241+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e0004eb0 con 0x7f38f402b0a0 2024-03-29T10:49:42.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.241+0000 7f38f8ff9640 10 client.?.objecter ms_handle_connect 0x7f38f40c9570 2024-03-29T10:49:42.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.241+0000 7f38f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:42.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.241+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805539010 0 0) 0x7f38f402ca20 con 0x7f38f4029eb0 2024-03-29T10:49:42.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:42.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:42.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): my global_id is 4691 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f40cbca0 con 0x7f38f4029eb0 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 328701340 0 0) 0x7f38e0004eb0 con 0x7f38f402b0a0 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38f402ca20 con 0x7f38f402b0a0 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220119924 0 0) 0x7f38f402c890 con 0x7f38f40c9570 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): my global_id is 4552 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e0004eb0 con 0x7f38f40c9570 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f38f00018d0 con 0x7f38f402b0a0 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:42.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): dump: 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.245+0000 7f38f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:42.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 84249264 0 0) 0x7f38f402ca20 con 0x7f38f402b0a0 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 >> v1:172.21.15.114:6790/0 conn(0x7f38f4029eb0 legacy=0x7f38f40c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 >> v1:172.21.15.176:6789/0 conn(0x7f38f40c9570 legacy=0x7f38f40cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f38f8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f38f8ff9640 1 monclient: found mon.a 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f38f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f38f8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f40c4cb0 con 0x7f38f402b0a0 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f38f8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f38f8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f38f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f39015ad640 5 monclient: authenticate success, global_id 4593 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f39015ad640 10 monclient: _renew_subs 2024-03-29T10:49:42.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f39015ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f39015ad640 1 -- 172.21.15.114:0/445935815 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f40c4f60 con 0x7f38f402b0a0 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f39015ad640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f39015ad640 10 monclient: _renew_subs 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f39015ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f39015ad640 1 -- 172.21.15.114:0/445935815 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f40ceba0 con 0x7f38f402b0a0 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.246+0000 7f39015ad640 1 librados: init done 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.248+0000 7f38e67fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.248+0000 7f38e67fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.248+0000 7f38e67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.248+0000 7f38e67fc640 1 -- 172.21.15.114:0/445935815 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f402ad00 con 0x7f38f402b0a0 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.249+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38f0003610 con 0x7f38f402b0a0 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.249+0000 7f38f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.249+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f38f0003a20 con 0x7f38f402b0a0 2024-03-29T10:49:42.235 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.249+0000 7f38f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.249+0000 7f38f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.249+0000 7f38f8ff9640 10 monclient: dump: 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:42.236 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.249+0000 7f38f9ffb640 4 set_mon_vals no callback set 2024-03-29T10:49:42.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.250+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f38f0003c40 con 0x7f38f402b0a0 2024-03-29T10:49:42.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.251+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1764351048 0 0) 0x7f38f0084ad0 con 0x7f38f402b0a0 2024-03-29T10:49:42.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.251+0000 7f38f8ff9640 10 client.4593.objecter ms_dispatch 0x7f38fc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:42.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.251+0000 7f38f8ff9640 3 client.4593.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:42.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.251+0000 7f38f8ff9640 3 client.4593.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:42.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.251+0000 7f38f8ff9640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-29T10:49:42.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.251+0000 7f38fbfff640 10 monclient: get_auth_request con 0x7f38e006db40 auth_method 0 2024-03-29T10:49:42.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.251+0000 7f38f8ff9640 10 client.4593.objecter ms_handle_connect 0x7f38e006db40 2024-03-29T10:49:42.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.259+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38f00503b0 con 0x7f38f402b0a0 2024-03-29T10:49:42.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.260+0000 7f38f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:42.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.260+0000 7f38f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:42.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.590+0000 7f38e67fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:42.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.590+0000 7f38e67fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:42.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.590+0000 7f38e67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:49:42.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.590+0000 7f38e67fc640 1 -- 172.21.15.114:0/445935815 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38f40cf7b0 con 0x7f38f402b0a0 2024-03-29T10:49:42.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.595+0000 7f38f8ff9640 1 -- 172.21.15.114:0/445935815 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2201894833) 0x7f38f00522e0 con 0x7f38f402b0a0 2024-03-29T10:49:42.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.595+0000 7f38f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:42.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.595+0000 7f38f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:42.578 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:49:42.578 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:49:18.930439+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:18.930439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:42.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.599+0000 7f38e67fc640 10 librados: watch_flush enter 2024-03-29T10:49:42.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.600+0000 7f38e67fc640 10 librados: watch_flush exit 2024-03-29T10:49:42.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.600+0000 7f38e67fc640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-29T10:49:42.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.600+0000 7f38e67fc640 10 client.4593.objecter successfully canceled tick 2024-03-29T10:49:42.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.600+0000 7f38e67fc640 1 -- 172.21.15.114:0/445935815 >> v1:172.21.15.114:6816/775044307 conn(0x7f38e006db40 legacy=0x7f38e0070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.600+0000 7f38e67fc640 10 monclient: shutdown 2024-03-29T10:49:42.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.600+0000 7f38e67fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:42.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.600+0000 7f38e67fc640 1 -- 172.21.15.114:0/445935815 >> v1:172.21.15.114:6789/0 conn(0x7f38f402b0a0 legacy=0x7f38f40cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.600+0000 7f38e67fc640 1 -- 172.21.15.114:0/445935815 shutdown_connections 2024-03-29T10:49:42.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.600+0000 7f38e67fc640 1 -- 172.21.15.114:0/445935815 >> 172.21.15.114:0/445935815 conn(0x7f38f40058f0 msgr2=0x7f38f40373e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:42.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.601+0000 7f38e67fc640 1 -- 172.21.15.114:0/445935815 shutdown_connections 2024-03-29T10:49:42.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.601+0000 7f38e67fc640 1 -- 172.21.15.114:0/445935815 wait complete. 2024-03-29T10:49:42.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:42.602+0000 7f38e67fc640 1 librados: shutdown 2024-03-29T10:49:42.600 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:49.407 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:49.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.572+0000 7fdd0f59e640 10 monclient: get_monmap_and_config 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.572+0000 7fdd0f59e640 10 monclient: build_initial_monmap 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.572+0000 7fdd0f59e640 10 monclient: monmap: 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:49:49.573997+0000 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:49:49.573997+0000 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:49:49.556 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:49.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.575+0000 7fdd0f59e640 1 Processor -- start 2024-03-29T10:49:49.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.575+0000 7fdd0f59e640 1 -- start start 2024-03-29T10:49:49.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.575+0000 7fdd0f59e640 10 monclient: init 2024-03-29T10:49:49.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.576+0000 7fdd0f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:49.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.576+0000 7fdd0f59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:49.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.576+0000 7fdd0f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fdd100a9750 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:49.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.576+0000 7fdd0f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fdd100adca0 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:49.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.578+0000 7fdd0f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fdd10146a80 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:49.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.578+0000 7fdd0f59e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd100abdc0 con 0x7fdd10146a80 2024-03-29T10:49:49.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.578+0000 7fdd0f59e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd100b0070 con 0x7fdd100a9750 2024-03-29T10:49:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.578+0000 7fdd0f59e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd100af880 con 0x7fdd100adca0 2024-03-29T10:49:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.578+0000 7fdd0f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.578+0000 7fdd0f59e640 10 monclient(hunting): authenticate will time out at 1454.819538s 2024-03-29T10:49:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0dd9b640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fdd100adca0 0x7fdd10146370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42780/0 (socket says 172.21.15.114:42780) 2024-03-29T10:49:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0dd9b640 1 -- 172.21.15.114:0/2382710001 learned_addr learned my addr 172.21.15.114:0/2382710001 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 1 -- 172.21.15.114:0/2382710001 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911097721 0 0) 0x7fdd100af880 con 0x7fdd100adca0 2024-03-29T10:49:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 10 monclient(hunting): my global_id is 4694 2024-03-29T10:49:49.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 1 -- 172.21.15.114:0/2382710001 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf8003520 con 0x7fdd100adca0 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 1 -- 172.21.15.114:0/2382710001 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473422289 0 0) 0x7fdd100abdc0 con 0x7fdd10146a80 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 10 monclient(hunting): my global_id is 4596 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.579+0000 7fdd0d59a640 1 -- 172.21.15.114:0/2382710001 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd100af880 con 0x7fdd10146a80 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 1 -- 172.21.15.114:0/2382710001 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1771717657 0 0) 0x7fdcf8003520 con 0x7fdd100adca0 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 1 -- 172.21.15.114:0/2382710001 >> v1:172.21.15.176:6789/0 conn(0x7fdd100a9750 legacy=0x7fdd10143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 1 -- 172.21.15.114:0/2382710001 >> v1:172.21.15.114:6789/0 conn(0x7fdd10146a80 legacy=0x7fdd10151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 1 monclient: found mon.noname-b 2024-03-29T10:49:49.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 1 -- 172.21.15.114:0/2382710001 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd100ac030 con 0x7fdd100adca0 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 10 monclient: _finish_auth 0 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 1 -- 172.21.15.114:0/2382710001 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdd04001690 con 0x7fdd100adca0 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 10 monclient: dump: 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:49.566 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 1 -- 172.21.15.114:0/2382710001 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd040017f0 con 0x7fdd100adca0 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 1 -- 172.21.15.114:0/2382710001 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdd04001c00 con 0x7fdd100adca0 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.580+0000 7fdd0d59a640 10 monclient: dump: 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:49.567 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:49.568 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:49.568 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:49.568 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:49.568 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:49.568 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:49.568 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:49.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.581+0000 7fdd0f59e640 5 monclient: authenticate success, global_id 4694 2024-03-29T10:49:49.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.581+0000 7fdd0f59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:49.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.581+0000 7fdd0f59e640 4 set_mon_vals no callback set 2024-03-29T10:49:49.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.582+0000 7fdd0f59e640 10 monclient: shutdown 2024-03-29T10:49:49.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.582+0000 7fdd0f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:49.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.582+0000 7fdd0f59e640 1 -- 172.21.15.114:0/2382710001 >> v1:172.21.15.114:6790/0 conn(0x7fdd100adca0 legacy=0x7fdd10146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.583+0000 7fdd0f59e640 1 -- 172.21.15.114:0/2382710001 shutdown_connections 2024-03-29T10:49:49.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.583+0000 7fdd0f59e640 1 -- 172.21.15.114:0/2382710001 >> 172.21.15.114:0/2382710001 conn(0x7fdd100673e0 msgr2=0x7fdd100b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:49.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.585+0000 7fdd0f59e640 1 -- 172.21.15.114:0/2382710001 shutdown_connections 2024-03-29T10:49:49.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.585+0000 7fdd0f59e640 1 -- 172.21.15.114:0/2382710001 wait complete. 2024-03-29T10:49:49.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.586+0000 7fdd0f59e640 10 monclient: build_initial_monmap 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.586+0000 7fdd0f59e640 10 monclient: monmap: 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.587+0000 7fdd0f59e640 1 librados: starting msgr at 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.587+0000 7fdd0f59e640 1 librados: starting objecter 2024-03-29T10:49:49.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.587+0000 7fdd0f59e640 1 Processor -- start 2024-03-29T10:49:49.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.587+0000 7fdd0f59e640 1 -- start start 2024-03-29T10:49:49.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.587+0000 7fdd0f59e640 1 librados: setting wanted keys 2024-03-29T10:49:49.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.587+0000 7fdd0f59e640 1 librados: calling monclient init 2024-03-29T10:49:49.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.587+0000 7fdd0f59e640 10 monclient: init 2024-03-29T10:49:49.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.588+0000 7fdd0f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:49.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.588+0000 7fdd0f59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:49.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.590+0000 7fdd0f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fdd100a9750 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:49.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.590+0000 7fdd0f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fdd100adca0 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:49.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.590+0000 7fdd0f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fdd10146a80 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:49.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.590+0000 7fdd0f59e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd10146390 con 0x7fdd10146a80 2024-03-29T10:49:49.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.590+0000 7fdd0f59e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd10072710 con 0x7fdd100a9750 2024-03-29T10:49:49.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.590+0000 7fdd0f59e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd10144170 con 0x7fdd100adca0 2024-03-29T10:49:49.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.590+0000 7fdd0f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:49.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.590+0000 7fdd0f59e640 10 monclient(hunting): authenticate will time out at 1454.831110s 2024-03-29T10:49:49.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.591+0000 7fdd0dd9b640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fdd100adca0 0x7fdd101db3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:43654/0 (socket says 172.21.15.114:43654) 2024-03-29T10:49:49.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.591+0000 7fdd0dd9b640 1 -- 172.21.15.114:0/1436351119 learned_addr learned my addr 172.21.15.114:0/1436351119 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:49.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.591+0000 7fdcfeffd640 10 client.?.objecter ms_handle_connect 0x7fdd100adca0 2024-03-29T10:49:49.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.591+0000 7fdcfeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:49.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.591+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103439367 0 0) 0x7fdd10144170 con 0x7fdd100adca0 2024-03-29T10:49:49.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.591+0000 7fdcfeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:49.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.591+0000 7fdcfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:49.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.591+0000 7fdcfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.591+0000 7fdcfeffd640 10 monclient(hunting): my global_id is 4697 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.591+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf0004eb0 con 0x7fdd100adca0 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1090538977 0 0) 0x7fdcf0004eb0 con 0x7fdd100adca0 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcf0005300 con 0x7fdd100adca0 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdd04001c00 con 0x7fdd100adca0 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 10 monclient(hunting): dump: 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:49.576 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:49.577 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:49.577 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:49.577 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:49.577 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:49.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:49.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 10 client.?.objecter ms_handle_connect 0x7fdd100a9750 2024-03-29T10:49:49.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:49.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3881314963 0 0) 0x7fdcf0005300 con 0x7fdd100adca0 2024-03-29T10:49:49.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 >> v1:172.21.15.176:6789/0 conn(0x7fdd100a9750 legacy=0x7fdd1013e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.592+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 >> v1:172.21.15.114:6789/0 conn(0x7fdd10146a80 legacy=0x7fdd101dbb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:49.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.593+0000 7fdcfeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:49.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.593+0000 7fdcfeffd640 1 monclient: found mon.c 2024-03-29T10:49:49.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.593+0000 7fdcfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:49.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.593+0000 7fdcfeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:49.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.593+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd10144300 con 0x7fdd100adca0 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.593+0000 7fdcfeffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.593+0000 7fdcfeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.593+0000 7fdcfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdd0f59e640 5 monclient: authenticate success, global_id 4697 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdd0f59e640 10 monclient: _renew_subs 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd04004b10 con 0x7fdd100adca0 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdd0f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdd0f59e640 1 -- 172.21.15.114:0/1436351119 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd10071c00 con 0x7fdd100adca0 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdcfeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7fdd04001c00 con 0x7fdd100adca0 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdcfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdcfeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:49.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdcfeffd640 10 monclient: dump: 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdd0f59e640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdd0f59e640 10 monclient: _renew_subs 2024-03-29T10:49:49.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdd0f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:49.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdd0f59e640 1 -- 172.21.15.114:0/1436351119 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1013e810 con 0x7fdd100adca0 2024-03-29T10:49:49.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.594+0000 7fdd0f59e640 1 librados: init done 2024-03-29T10:49:49.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.596+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7fdd04004e40 con 0x7fdd100adca0 2024-03-29T10:49:49.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.597+0000 7fdd0f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:49.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.597+0000 7fdd0cd99640 4 set_mon_vals no callback set 2024-03-29T10:49:49.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.597+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1764351048 0 0) 0x7fdd04084a40 con 0x7fdd100adca0 2024-03-29T10:49:49.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.597+0000 7fdcfeffd640 10 client.4697.objecter ms_dispatch 0x7fdd10000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:49.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.597+0000 7fdcfeffd640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:49.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.597+0000 7fdcfeffd640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:49.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.597+0000 7fdcfeffd640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-29T10:49:49.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.598+0000 7fdd0f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:49.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.598+0000 7fdd0f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:49.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.598+0000 7fdd0f59e640 1 -- 172.21.15.114:0/1436351119 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1004f730 con 0x7fdd100adca0 2024-03-29T10:49:49.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.599+0000 7fdd0e59c640 10 monclient: get_auth_request con 0x7fdcf006e0d0 auth_method 0 2024-03-29T10:49:49.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.599+0000 7fdcfeffd640 10 client.4697.objecter ms_handle_connect 0x7fdcf006e0d0 2024-03-29T10:49:49.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.607+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd04050320 con 0x7fdd100adca0 2024-03-29T10:49:49.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.607+0000 7fdcfeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:49.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.607+0000 7fdcfeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:49.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.938+0000 7fdd0f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:49.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.938+0000 7fdd0f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:49.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.938+0000 7fdd0f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T10:49:49.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.938+0000 7fdd0f59e640 1 -- 172.21.15.114:0/1436351119 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd10146540 con 0x7fdd100adca0 2024-03-29T10:49:49.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.940+0000 7fdcfeffd640 1 -- 172.21.15.114:0/1436351119 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2201894833) 0x7fdd10146540 con 0x7fdd100adca0 2024-03-29T10:49:49.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.940+0000 7fdcfeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:49.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.940+0000 7fdcfeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:49.922 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:49:49.923 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:49:18.930439+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:18.930439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:49.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.943+0000 7fdd0f59e640 10 librados: watch_flush enter 2024-03-29T10:49:49.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.943+0000 7fdd0f59e640 10 librados: watch_flush exit 2024-03-29T10:49:49.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.943+0000 7fdd0f59e640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-29T10:49:49.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.944+0000 7fdd0f59e640 10 client.4697.objecter successfully canceled tick 2024-03-29T10:49:49.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.944+0000 7fdd0f59e640 1 -- 172.21.15.114:0/1436351119 >> v1:172.21.15.114:6816/775044307 conn(0x7fdcf006e0d0 legacy=0x7fdcf0070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.944+0000 7fdd0f59e640 10 monclient: shutdown 2024-03-29T10:49:49.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.944+0000 7fdd0f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:49.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.944+0000 7fdd0f59e640 1 -- 172.21.15.114:0/1436351119 >> v1:172.21.15.114:6790/0 conn(0x7fdd100adca0 legacy=0x7fdd101db3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.944+0000 7fdd0f59e640 1 -- 172.21.15.114:0/1436351119 shutdown_connections 2024-03-29T10:49:49.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.944+0000 7fdd0f59e640 1 -- 172.21.15.114:0/1436351119 >> 172.21.15.114:0/1436351119 conn(0x7fdd100673e0 msgr2=0x7fdd100aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:49.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.944+0000 7fdd0f59e640 1 -- 172.21.15.114:0/1436351119 shutdown_connections 2024-03-29T10:49:49.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.944+0000 7fdd0f59e640 1 -- 172.21.15.114:0/1436351119 wait complete. 2024-03-29T10:49:49.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:49.944+0000 7fdd0f59e640 1 librados: shutdown 2024-03-29T10:49:49.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:56.751 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:56.910 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.926+0000 7f11deb39640 10 monclient: get_monmap_and_config 2024-03-29T10:49:56.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.926+0000 7f11deb39640 10 monclient: build_initial_monmap 2024-03-29T10:49:56.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.926+0000 7f11deb39640 10 monclient: monmap: 2024-03-29T10:49:56.911 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:56.911 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:56.911 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:49:56.927046+0000 2024-03-29T10:49:56.911 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:49:56.927046+0000 2024-03-29T10:49:56.911 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:56.911 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:56.911 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:56.912 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:49:56.912 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:49:56.912 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:56.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.928+0000 7f11deb39640 1 Processor -- start 2024-03-29T10:49:56.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.928+0000 7f11deb39640 1 -- start start 2024-03-29T10:49:56.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.928+0000 7f11deb39640 10 monclient: init 2024-03-29T10:49:56.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.928+0000 7f11deb39640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:56.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.928+0000 7f11deb39640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:56.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.929+0000 7f11deb39640 10 monclient(hunting): picked mon.noname-b con 0x7f11d80ad6c0 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:56.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.929+0000 7f11deb39640 10 monclient(hunting): picked mon.noname-a con 0x7f11d80af6c0 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:56.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.929+0000 7f11deb39640 10 monclient(hunting): picked mon.noname-c con 0x7f11d814fdc0 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:56.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.929+0000 7f11deb39640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d80a94e0 con 0x7f11d80af6c0 2024-03-29T10:49:56.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.929+0000 7f11deb39640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d80af3a0 con 0x7f11d814fdc0 2024-03-29T10:49:56.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.929+0000 7f11deb39640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d80afb60 con 0x7f11d80ad6c0 2024-03-29T10:49:56.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.929+0000 7f11deb39640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:56.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.929+0000 7f11deb39640 10 monclient(hunting): authenticate will time out at 1462.169108s 2024-03-29T10:49:56.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.929+0000 7f11dd0d6640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f11d814fdc0 0x7f11d815a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:44180/0 (socket says 172.21.15.114:44180) 2024-03-29T10:49:56.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.929+0000 7f11dd0d6640 1 -- 172.21.15.114:0/3565533638 learned_addr learned my addr 172.21.15.114:0/3565533638 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:56.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.930+0000 7f11d77fe640 1 -- 172.21.15.114:0/3565533638 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310140140 0 0) 0x7f11d80af3a0 con 0x7f11d814fdc0 2024-03-29T10:49:56.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.930+0000 7f11d77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:56.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.930+0000 7f11d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:56.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.930+0000 7f11d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:56.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.930+0000 7f11d77fe640 10 monclient(hunting): my global_id is 4558 2024-03-29T10:49:56.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.930+0000 7f11d77fe640 1 -- 172.21.15.114:0/3565533638 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c0003520 con 0x7f11d814fdc0 2024-03-29T10:49:56.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.932+0000 7f11d77fe640 1 -- 172.21.15.114:0/3565533638 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3979145329 0 0) 0x7f11c0003520 con 0x7f11d814fdc0 2024-03-29T10:49:56.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.932+0000 7f11d77fe640 1 -- 172.21.15.114:0/3565533638 >> v1:172.21.15.114:6790/0 conn(0x7f11d80ad6c0 legacy=0x7f11d814f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:56.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.932+0000 7f11d77fe640 1 -- 172.21.15.114:0/3565533638 >> v1:172.21.15.114:6789/0 conn(0x7f11d80af6c0 legacy=0x7f11d8152980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:56.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:56.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 1 monclient: found mon.noname-c 2024-03-29T10:49:56.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:56.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T10:49:56.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 1 -- 172.21.15.114:0/3565533638 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d80aab80 con 0x7f11d814fdc0 2024-03-29T10:49:56.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:56.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:56.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:56.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 1 -- 172.21.15.114:0/3565533638 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f11d00012f0 con 0x7f11d814fdc0 2024-03-29T10:49:56.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:56.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:56.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.933+0000 7f11d77fe640 10 monclient: dump: 2024-03-29T10:49:56.922 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:56.922 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:56.923 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:56.923 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:56.923 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:56.923 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:56.923 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:56.923 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:56.924 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:56.924 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:56.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.934+0000 7f11d77fe640 1 -- 172.21.15.114:0/3565533638 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11d00014c0 con 0x7f11d814fdc0 2024-03-29T10:49:56.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.934+0000 7f11d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:56.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.934+0000 7f11d77fe640 1 -- 172.21.15.114:0/3565533638 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f11d0001910 con 0x7f11d814fdc0 2024-03-29T10:49:56.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.934+0000 7f11d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:56.925 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.934+0000 7f11d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:56.925 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.934+0000 7f11d77fe640 10 monclient: dump: 2024-03-29T10:49:56.925 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:56.925 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:56.925 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:56.925 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:56.925 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:56.925 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:56.926 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:56.926 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:56.926 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:56.926 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:56.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.935+0000 7f11deb39640 5 monclient: authenticate success, global_id 4558 2024-03-29T10:49:56.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.935+0000 7f11deb39640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:56.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.935+0000 7f11deb39640 4 set_mon_vals no callback set 2024-03-29T10:49:56.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.936+0000 7f11deb39640 10 monclient: shutdown 2024-03-29T10:49:56.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.936+0000 7f11deb39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:56.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.936+0000 7f11deb39640 1 -- 172.21.15.114:0/3565533638 >> v1:172.21.15.176:6789/0 conn(0x7f11d814fdc0 legacy=0x7f11d815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:56.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.936+0000 7f11deb39640 1 -- 172.21.15.114:0/3565533638 shutdown_connections 2024-03-29T10:49:56.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.936+0000 7f11deb39640 1 -- 172.21.15.114:0/3565533638 >> 172.21.15.114:0/3565533638 conn(0x7f11d80b35e0 msgr2=0x7f11d80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:56.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.938+0000 7f11deb39640 1 -- 172.21.15.114:0/3565533638 shutdown_connections 2024-03-29T10:49:56.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.938+0000 7f11deb39640 1 -- 172.21.15.114:0/3565533638 wait complete. 2024-03-29T10:49:56.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.939+0000 7f11deb39640 10 monclient: build_initial_monmap 2024-03-29T10:49:56.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.939+0000 7f11deb39640 10 monclient: monmap: 2024-03-29T10:49:56.932 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:49:56.932 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:56.932 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:49:56.932 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:49:56.932 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:56.932 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:56.932 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:49:56.932 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.939+0000 7f11deb39640 1 librados: starting msgr at 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.939+0000 7f11deb39640 1 librados: starting objecter 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.939+0000 7f11deb39640 1 Processor -- start 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.939+0000 7f11deb39640 1 -- start start 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.939+0000 7f11deb39640 1 librados: setting wanted keys 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.939+0000 7f11deb39640 1 librados: calling monclient init 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.939+0000 7f11deb39640 10 monclient: init 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.940+0000 7f11deb39640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.940+0000 7f11deb39640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.940+0000 7f11deb39640 10 monclient(hunting): picked mon.noname-c con 0x7f11d80ad6c0 addr v1:172.21.15.114:6790/0 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.940+0000 7f11deb39640 10 monclient(hunting): picked mon.noname-a con 0x7f11d80af6c0 addr v1:172.21.15.114:6789/0 2024-03-29T10:49:56.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.940+0000 7f11deb39640 10 monclient(hunting): picked mon.noname-b con 0x7f11d814fdc0 addr v1:172.21.15.176:6789/0 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.941+0000 7f11deb39640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d807b9d0 con 0x7f11d80af6c0 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.941+0000 7f11deb39640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d807bcb0 con 0x7f11d814fdc0 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.941+0000 7f11deb39640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d807d700 con 0x7f11d80ad6c0 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.941+0000 7f11deb39640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.941+0000 7f11deb39640 10 monclient(hunting): authenticate will time out at 1462.181064s 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.944+0000 7f11dd0d6640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f11d814fdc0 0x7f11d8210a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:45812/0 (socket says 172.21.15.114:45812) 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.944+0000 7f11dd0d6640 1 -- 172.21.15.114:0/2252671089 learned_addr learned my addr 172.21.15.114:0/2252671089 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.944+0000 7f11d57fa640 10 client.?.objecter ms_handle_connect 0x7f11d814fdc0 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.944+0000 7f11d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.945+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949666310 0 0) 0x7f11d807bcb0 con 0x7f11d814fdc0 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.945+0000 7f11d57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.945+0000 7f11d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:56.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.945+0000 7f11d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.945+0000 7f11d57fa640 10 monclient(hunting): my global_id is 4561 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.945+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c4004eb0 con 0x7f11d814fdc0 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.946+0000 7f11d57fa640 10 client.?.objecter ms_handle_connect 0x7f11d80ad6c0 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.946+0000 7f11d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.947+0000 7f11d57fa640 10 client.?.objecter ms_handle_connect 0x7f11d80af6c0 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.947+0000 7f11d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.947+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435750044 0 0) 0x7f11d807b9d0 con 0x7f11d80af6c0 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.947+0000 7f11d57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.947+0000 7f11d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.947+0000 7f11d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.947+0000 7f11d57fa640 10 monclient(hunting): my global_id is 4599 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.947+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d807bcb0 con 0x7f11d80af6c0 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2904516880 0 0) 0x7f11d807bcb0 con 0x7f11d80af6c0 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11d807b9d0 con 0x7f11d80af6c0 2024-03-29T10:49:56.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f11cc001810 con 0x7f11d80af6c0 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 10 monclient(hunting): dump: 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:56.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1910476821 0 0) 0x7f11c4004eb0 con 0x7f11d814fdc0 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11d807bcb0 con 0x7f11d814fdc0 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f11d00017d0 con 0x7f11d814fdc0 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 10 monclient(hunting): dump: 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:56.937 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827846591 0 0) 0x7f11d807d700 con 0x7f11d80ad6c0 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 10 monclient(hunting): my global_id is 4700 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.948+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c4004eb0 con 0x7f11d80ad6c0 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1417419126 0 0) 0x7f11d807b9d0 con 0x7f11d80af6c0 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 >> v1:172.21.15.114:6790/0 conn(0x7f11d80ad6c0 legacy=0x7f11d807c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 >> v1:172.21.15.176:6789/0 conn(0x7f11d814fdc0 legacy=0x7f11d8210a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 1 monclient: found mon.a 2024-03-29T10:49:56.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d807d9e0 con 0x7f11d80af6c0 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 10 monclient: _finish_auth 0 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11cc001d00 con 0x7f11d80af6c0 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f11cc003790 con 0x7f11d80af6c0 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.949+0000 7f11d57fa640 10 monclient: dump: 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:56.939 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.950+0000 7f11deb39640 5 monclient: authenticate success, global_id 4599 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.950+0000 7f11deb39640 10 monclient: _renew_subs 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.950+0000 7f11deb39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.950+0000 7f11deb39640 1 -- 172.21.15.114:0/2252671089 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d807b660 con 0x7f11d80af6c0 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.950+0000 7f11deb39640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.950+0000 7f11deb39640 10 monclient: _renew_subs 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.950+0000 7f11deb39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.950+0000 7f11deb39640 1 -- 172.21.15.114:0/2252671089 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d8070990 con 0x7f11d80af6c0 2024-03-29T10:49:56.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.950+0000 7f11deb39640 1 librados: init done 2024-03-29T10:49:56.942 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.951+0000 7f11d6ffd640 4 set_mon_vals no callback set 2024-03-29T10:49:56.942 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.952+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f11cc019760 con 0x7f11d80af6c0 2024-03-29T10:49:56.942 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.953+0000 7f11deb39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:56.942 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.953+0000 7f11deb39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:56.942 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.953+0000 7f11deb39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:49:56.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.953+0000 7f11deb39640 1 -- 172.21.15.114:0/2252671089 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b8005080 con 0x7f11d80af6c0 2024-03-29T10:49:56.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.953+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1764351048 0 0) 0x7f11cc0837b0 con 0x7f11d80af6c0 2024-03-29T10:49:56.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.953+0000 7f11d57fa640 10 client.4599.objecter ms_dispatch 0x7f11d8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:56.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.953+0000 7f11d57fa640 3 client.4599.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:56.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.953+0000 7f11d57fa640 3 client.4599.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:56.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.953+0000 7f11d57fa640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-29T10:49:56.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.953+0000 7f11dc8d5640 10 monclient: get_auth_request con 0x7f11c406df90 auth_method 0 2024-03-29T10:49:56.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.954+0000 7f11d57fa640 10 client.4599.objecter ms_handle_connect 0x7f11c406df90 2024-03-29T10:49:56.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.965+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11cc04eec0 con 0x7f11d80af6c0 2024-03-29T10:49:56.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.965+0000 7f11d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:56.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:56.965+0000 7f11d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:57.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.301+0000 7f11deb39640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:57.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.301+0000 7f11deb39640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:57.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.301+0000 7f11deb39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:49:57.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.301+0000 7f11deb39640 1 -- 172.21.15.114:0/2252671089 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11b8005330 con 0x7f11d80af6c0 2024-03-29T10:49:57.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.304+0000 7f11d57fa640 1 -- 172.21.15.114:0/2252671089 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2201894833) 0x7f11cc050fc0 con 0x7f11d80af6c0 2024-03-29T10:49:57.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.304+0000 7f11d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:57.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.304+0000 7f11d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:57.286 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:49:57.286 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:49:18.930439+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:18.930439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:57.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.308+0000 7f11deb39640 10 librados: watch_flush enter 2024-03-29T10:49:57.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.309+0000 7f11deb39640 10 librados: watch_flush exit 2024-03-29T10:49:57.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.309+0000 7f11deb39640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-29T10:49:57.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.309+0000 7f11deb39640 10 client.4599.objecter successfully canceled tick 2024-03-29T10:49:57.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.309+0000 7f11deb39640 1 -- 172.21.15.114:0/2252671089 >> v1:172.21.15.114:6816/775044307 conn(0x7f11c406df90 legacy=0x7f11c4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:57.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.309+0000 7f11deb39640 10 monclient: shutdown 2024-03-29T10:49:57.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.309+0000 7f11deb39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:57.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.309+0000 7f11deb39640 1 -- 172.21.15.114:0/2252671089 >> v1:172.21.15.114:6789/0 conn(0x7f11d80af6c0 legacy=0x7f11d8070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:57.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.310+0000 7f11deb39640 1 -- 172.21.15.114:0/2252671089 shutdown_connections 2024-03-29T10:49:57.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.310+0000 7f11deb39640 1 -- 172.21.15.114:0/2252671089 >> 172.21.15.114:0/2252671089 conn(0x7f11d80b35e0 msgr2=0x7f11d80ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:57.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.310+0000 7f11deb39640 1 -- 172.21.15.114:0/2252671089 shutdown_connections 2024-03-29T10:49:57.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.310+0000 7f11deb39640 1 -- 172.21.15.114:0/2252671089 wait complete. 2024-03-29T10:49:57.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:49:57.310+0000 7f11deb39640 1 librados: shutdown 2024-03-29T10:49:57.308 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:04.115 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:04.262 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.280+0000 7f723a4f1640 10 monclient: get_monmap_and_config 2024-03-29T10:50:04.262 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.280+0000 7f723a4f1640 10 monclient: build_initial_monmap 2024-03-29T10:50:04.262 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.280+0000 7f723a4f1640 10 monclient: monmap: 2024-03-29T10:50:04.262 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:50:04.263 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:50:04.263 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:50:04.281361+0000 2024-03-29T10:50:04.263 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:50:04.281361+0000 2024-03-29T10:50:04.263 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:04.263 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:04.263 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:50:04.263 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:50:04.263 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:50:04.263 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:04.264 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.281+0000 7f723a4f1640 1 Processor -- start 2024-03-29T10:50:04.264 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.281+0000 7f723a4f1640 1 -- start start 2024-03-29T10:50:04.264 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.281+0000 7f723a4f1640 10 monclient: init 2024-03-29T10:50:04.264 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f723a4f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:04.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f723a4f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:04.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f723a4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f72341457f0 addr v1:172.21.15.176:6789/0 2024-03-29T10:50:04.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f723a4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f723413f650 addr v1:172.21.15.114:6789/0 2024-03-29T10:50:04.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f723a4f1640 10 monclient(hunting): picked mon.noname-b con 0x7f72340a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T10:50:04.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f7233fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f72341457f0 0x7f723413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:46800/0 (socket says 172.21.15.114:46800) 2024-03-29T10:50:04.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f7233fff640 1 -- 172.21.15.114:0/267375904 learned_addr learned my addr 172.21.15.114:0/267375904 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:50:04.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f723a4f1640 1 -- 172.21.15.114:0/267375904 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72340aaef0 con 0x7f723413f650 2024-03-29T10:50:04.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f723a4f1640 1 -- 172.21.15.114:0/267375904 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7234145bd0 con 0x7f72341457f0 2024-03-29T10:50:04.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f723a4f1640 1 -- 172.21.15.114:0/267375904 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7234145eb0 con 0x7f72340a9d70 2024-03-29T10:50:04.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f723a4f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:04.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.282+0000 7f723a4f1640 10 monclient(hunting): authenticate will time out at 1469.523512s 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 1 -- 172.21.15.114:0/267375904 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233943778 0 0) 0x7f72340aaef0 con 0x7f723413f650 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 10 monclient(hunting): my global_id is 4602 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 1 -- 172.21.15.114:0/267375904 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f721c003520 con 0x7f723413f650 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 1 -- 172.21.15.114:0/267375904 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476815370 0 0) 0x7f7234145eb0 con 0x7f72340a9d70 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 10 monclient(hunting): my global_id is 4703 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 1 -- 172.21.15.114:0/267375904 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72340aaef0 con 0x7f72340a9d70 2024-03-29T10:50:04.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 1 -- 172.21.15.114:0/267375904 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1784117454 0 0) 0x7f721c003520 con 0x7f723413f650 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 1 -- 172.21.15.114:0/267375904 >> v1:172.21.15.114:6790/0 conn(0x7f72340a9d70 legacy=0x7f723414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 1 -- 172.21.15.114:0/267375904 >> v1:172.21.15.176:6789/0 conn(0x7f72341457f0 legacy=0x7f723413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 1 monclient: found mon.noname-a 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.283+0000 7f7232ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f7232ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f7232ffd640 1 -- 172.21.15.114:0/267375904 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7234146340 con 0x7f723413f650 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f7232ffd640 10 monclient: _finish_auth 0 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f7232ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f7232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f723a4f1640 5 monclient: authenticate success, global_id 4602 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f723a4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f7232ffd640 1 -- 172.21.15.114:0/267375904 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7224001a10 con 0x7f723413f650 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f7232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:04.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f7232ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f7232ffd640 10 monclient: dump: 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f723a4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:04.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f7232ffd640 1 -- 172.21.15.114:0/267375904 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7224001c30 con 0x7f723413f650 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f7232ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f723a4f1640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.284+0000 7f723a4f1640 4 set_mon_vals no callback set 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.285+0000 7f7232ffd640 1 -- 172.21.15.114:0/267375904 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7224001fc0 con 0x7f723413f650 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.285+0000 7f7232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.285+0000 7f7232ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.285+0000 7f7232ffd640 10 monclient: dump: 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:04.271 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:50:04.272 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:50:04.272 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:50:04.272 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:04.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.285+0000 7f723a4f1640 10 monclient: shutdown 2024-03-29T10:50:04.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.285+0000 7f723a4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:04.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.285+0000 7f723a4f1640 1 -- 172.21.15.114:0/267375904 >> v1:172.21.15.114:6789/0 conn(0x7f723413f650 legacy=0x7f72340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.285+0000 7f723a4f1640 1 -- 172.21.15.114:0/267375904 shutdown_connections 2024-03-29T10:50:04.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.285+0000 7f723a4f1640 1 -- 172.21.15.114:0/267375904 >> 172.21.15.114:0/267375904 conn(0x7f7234066a00 msgr2=0x7f7234066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:04.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.286+0000 7f723a4f1640 1 -- 172.21.15.114:0/267375904 shutdown_connections 2024-03-29T10:50:04.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.286+0000 7f723a4f1640 1 -- 172.21.15.114:0/267375904 wait complete. 2024-03-29T10:50:04.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.287+0000 7f723a4f1640 10 monclient: build_initial_monmap 2024-03-29T10:50:04.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.287+0000 7f723a4f1640 10 monclient: monmap: 2024-03-29T10:50:04.277 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:50:04.277 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:04.277 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:50:04.277 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:50:04.277 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.287+0000 7f723a4f1640 1 librados: starting msgr at 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.287+0000 7f723a4f1640 1 librados: starting objecter 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.287+0000 7f723a4f1640 1 Processor -- start 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.288+0000 7f723a4f1640 1 -- start start 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.288+0000 7f723a4f1640 1 librados: setting wanted keys 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.288+0000 7f723a4f1640 1 librados: calling monclient init 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.288+0000 7f723a4f1640 10 monclient: init 2024-03-29T10:50:04.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.288+0000 7f723a4f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.288+0000 7f723a4f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.288+0000 7f723a4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f72340a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.289+0000 7f7233fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f72340a9d70 0x7f7234139310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51092/0 (socket says 172.21.15.114:51092) 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.289+0000 7f7233fff640 1 -- 172.21.15.114:0/1996636301 learned_addr learned my addr 172.21.15.114:0/1996636301 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.289+0000 7f723a4f1640 10 monclient(hunting): picked mon.noname-b con 0x7f723413f650 addr v1:172.21.15.176:6789/0 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.289+0000 7f7230ff9640 10 client.?.objecter ms_handle_connect 0x7f72340a9d70 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.289+0000 7f7230ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.289+0000 7f723a4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f72341457f0 addr v1:172.21.15.114:6790/0 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.289+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723413abb0 con 0x7f72340a9d70 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.289+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7234135b30 con 0x7f723413f650 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.289+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723413ae20 con 0x7f72341457f0 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.289+0000 7f723a4f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.289+0000 7f723a4f1640 10 monclient(hunting): authenticate will time out at 1469.530810s 2024-03-29T10:50:04.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.290+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592747343 0 0) 0x7f723413abb0 con 0x7f72340a9d70 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.290+0000 7f7230ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.290+0000 7f7230ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.290+0000 7f7230ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.290+0000 7f7230ff9640 10 monclient(hunting): my global_id is 4605 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.290+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7220004eb0 con 0x7f72340a9d70 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.290+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 356970357 0 0) 0x7f7220004eb0 con 0x7f72340a9d70 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.290+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f723413abb0 con 0x7f72340a9d70 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f72280017c0 con 0x7f72340a9d70 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient(hunting): dump: 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:04.280 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 client.?.objecter ms_handle_connect 0x7f72341457f0 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 client.?.objecter ms_handle_connect 0x7f723413f650 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508455072 0 0) 0x7f7234135b30 con 0x7f723413f650 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:04.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient(hunting): my global_id is 4564 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7220004eb0 con 0x7f723413f650 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3997916572 0 0) 0x7f723413abb0 con 0x7f72340a9d70 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 >> v1:172.21.15.114:6790/0 conn(0x7f72341457f0 legacy=0x7f7234135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 >> v1:172.21.15.176:6789/0 conn(0x7f723413f650 legacy=0x7f7234134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 1 monclient: found mon.a 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723413b100 con 0x7f72340a9d70 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient: _finish_auth 0 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f7230ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f723a4f1640 5 monclient: authenticate success, global_id 4605 2024-03-29T10:50:04.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f723a4f1640 10 monclient: _renew_subs 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f723a4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7234138df0 con 0x7f72340a9d70 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f723a4f1640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f723a4f1640 10 monclient: _renew_subs 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f723a4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.291+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7234139e50 con 0x7f72340a9d70 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.292+0000 7f723a4f1640 1 librados: init done 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.292+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7228001920 con 0x7f72340a9d70 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.292+0000 7f7230ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.292+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f7228004980 con 0x7f72340a9d70 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.292+0000 7f7230ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.292+0000 7f7230ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.292+0000 7f7230ff9640 10 monclient: dump: 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:50:04.283 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.292+0000 7f72327fc640 4 set_mon_vals no callback set 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.293+0000 7f723a4f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.293+0000 7f723a4f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.293+0000 7f723a4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:50:04.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.293+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7234075f10 con 0x7f72340a9d70 2024-03-29T10:50:04.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.296+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f7228004b60 con 0x7f72340a9d70 2024-03-29T10:50:04.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.296+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1764351048 0 0) 0x7f7228084b40 con 0x7f72340a9d70 2024-03-29T10:50:04.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.296+0000 7f7230ff9640 10 client.4605.objecter ms_dispatch 0x7f7234000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:50:04.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.296+0000 7f7230ff9640 3 client.4605.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:50:04.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.296+0000 7f7230ff9640 3 client.4605.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:50:04.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.296+0000 7f72337fe640 10 monclient: get_auth_request con 0x7f722006ddd0 auth_method 0 2024-03-29T10:50:04.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.297+0000 7f7230ff9640 20 client.4605.objecter dump_active .. 0 homeless 2024-03-29T10:50:04.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.297+0000 7f7230ff9640 10 client.4605.objecter ms_handle_connect 0x7f722006ddd0 2024-03-29T10:50:04.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.306+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7228050250 con 0x7f72340a9d70 2024-03-29T10:50:04.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.306+0000 7f7230ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:04.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.306+0000 7f7230ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:04.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.696+0000 7f723a4f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:04.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.696+0000 7f723a4f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:04.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.696+0000 7f723a4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:50:04.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.696+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7234074f50 con 0x7f72340a9d70 2024-03-29T10:50:04.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.697+0000 7f7230ff9640 1 -- 172.21.15.114:0/1996636301 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2201894833) 0x7f7228052350 con 0x7f72340a9d70 2024-03-29T10:50:04.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.697+0000 7f7230ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:04.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.697+0000 7f7230ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:04.680 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:50:04.680 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:49:18.930439+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:18.930439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:04.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.700+0000 7f723a4f1640 10 librados: watch_flush enter 2024-03-29T10:50:04.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.700+0000 7f723a4f1640 10 librados: watch_flush exit 2024-03-29T10:50:04.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.700+0000 7f723a4f1640 20 client.4605.objecter shutdown clearing up homeless session... 2024-03-29T10:50:04.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.700+0000 7f723a4f1640 10 client.4605.objecter successfully canceled tick 2024-03-29T10:50:04.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.701+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 >> v1:172.21.15.114:6816/775044307 conn(0x7f722006ddd0 legacy=0x7f7220070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.701+0000 7f723a4f1640 10 monclient: shutdown 2024-03-29T10:50:04.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.701+0000 7f723a4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:04.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.701+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 >> v1:172.21.15.114:6789/0 conn(0x7f72340a9d70 legacy=0x7f7234139310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.701+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 shutdown_connections 2024-03-29T10:50:04.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.701+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 >> 172.21.15.114:0/1996636301 conn(0x7f7234066a00 msgr2=0x7f72340b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:04.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.701+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 shutdown_connections 2024-03-29T10:50:04.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.701+0000 7f723a4f1640 1 -- 172.21.15.114:0/1996636301 wait complete. 2024-03-29T10:50:04.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:04.701+0000 7f723a4f1640 1 librados: shutdown 2024-03-29T10:50:04.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:11.505 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:11.648 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.665+0000 7f2a3b4a8640 10 monclient: get_monmap_and_config 2024-03-29T10:50:11.648 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.665+0000 7f2a3b4a8640 10 monclient: build_initial_monmap 2024-03-29T10:50:11.648 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.665+0000 7f2a3b4a8640 10 monclient: monmap: 2024-03-29T10:50:11.648 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:50:11.648 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:50:11.648 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:50:11.667065+0000 2024-03-29T10:50:11.648 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:50:11.667065+0000 2024-03-29T10:50:11.648 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:11.648 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:11.649 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:50:11.649 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T10:50:11.649 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T10:50:11.649 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:11.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.666+0000 7f2a3b4a8640 1 Processor -- start 2024-03-29T10:50:11.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.667+0000 7f2a3b4a8640 1 -- start start 2024-03-29T10:50:11.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.667+0000 7f2a3b4a8640 10 monclient: init 2024-03-29T10:50:11.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.668+0000 7f2a3b4a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:11.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.668+0000 7f2a3b4a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.668+0000 7f2a3b4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f2a341422b0 addr v1:172.21.15.176:6789/0 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.668+0000 7f2a3b4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f2a34142690 addr v1:172.21.15.114:6790/0 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.668+0000 7f2a3b4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f2a340b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.668+0000 7f2a3b4a8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a34142c80 con 0x7f2a340b17d0 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.668+0000 7f2a3b4a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a340aae40 con 0x7f2a341422b0 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a3b4a8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a3413fbd0 con 0x7f2a34142690 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a3b4a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a39244640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f2a341422b0 0x7f2a3413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:57210/0 (socket says 172.21.15.114:57210) 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a39244640 1 -- 172.21.15.114:0/512437598 learned_addr learned my addr 172.21.15.114:0/512437598 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a39a45640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2a340b17d0 0x7f2a340b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36814/0 (socket says 172.21.15.114:36814) 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.668+0000 7f2a38a43640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f2a34142690 0x7f2a340b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34700/0 (socket says 172.21.15.114:34700) 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a3b4a8640 10 monclient(hunting): authenticate will time out at 1476.910503s 2024-03-29T10:50:11.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535556590 0 0) 0x7f2a34142c80 con 0x7f2a340b17d0 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 10 monclient(hunting): my global_id is 4608 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a20003520 con 0x7f2a340b17d0 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420511055 0 0) 0x7f2a340aae40 con 0x7f2a341422b0 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 10 monclient(hunting): my global_id is 4567 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.669+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a200048a0 con 0x7f2a341422b0 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573389502 0 0) 0x7f2a3413fbd0 con 0x7f2a34142690 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient(hunting): my global_id is 4709 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a20004be0 con 0x7f2a34142690 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021480061 0 0) 0x7f2a20003520 con 0x7f2a340b17d0 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 >> v1:172.21.15.114:6790/0 conn(0x7f2a34142690 legacy=0x7f2a340b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 >> v1:172.21.15.176:6789/0 conn(0x7f2a341422b0 legacy=0x7f2a3413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 1 monclient: found mon.noname-a 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a34151940 con 0x7f2a340b17d0 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient: _finish_auth 0 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:11.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2a240015f0 con 0x7f2a340b17d0 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient: dump: 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:11.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a3b4a8640 5 monclient: authenticate success, global_id 4608 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a3b4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a24001750 con 0x7f2a340b17d0 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a3b4a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.670+0000 7f2a3b4a8640 4 set_mon_vals no callback set 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.671+0000 7f2a2bfff640 1 -- 172.21.15.114:0/512437598 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2a24001ba0 con 0x7f2a340b17d0 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.671+0000 7f2a2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.671+0000 7f2a2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.671+0000 7f2a2bfff640 10 monclient: dump: 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.657 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:11.658 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:50:11.658 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:50:11.658 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:50:11.658 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:11.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.671+0000 7f2a3b4a8640 10 monclient: shutdown 2024-03-29T10:50:11.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.671+0000 7f2a3b4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:11.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.671+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/512437598 >> v1:172.21.15.114:6789/0 conn(0x7f2a340b17d0 legacy=0x7f2a340b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.672+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/512437598 shutdown_connections 2024-03-29T10:50:11.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.672+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/512437598 >> 172.21.15.114:0/512437598 conn(0x7f2a34066a00 msgr2=0x7f2a34066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:11.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.672+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/512437598 shutdown_connections 2024-03-29T10:50:11.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.672+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/512437598 wait complete. 2024-03-29T10:50:11.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.673+0000 7f2a3b4a8640 10 monclient: build_initial_monmap 2024-03-29T10:50:11.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.673+0000 7f2a3b4a8640 10 monclient: monmap: 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.674+0000 7f2a3b4a8640 1 librados: starting msgr at 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.674+0000 7f2a3b4a8640 1 librados: starting objecter 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.674+0000 7f2a3b4a8640 1 Processor -- start 2024-03-29T10:50:11.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.674+0000 7f2a3b4a8640 1 -- start start 2024-03-29T10:50:11.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.674+0000 7f2a3b4a8640 1 librados: setting wanted keys 2024-03-29T10:50:11.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.674+0000 7f2a3b4a8640 1 librados: calling monclient init 2024-03-29T10:50:11.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.674+0000 7f2a3b4a8640 10 monclient: init 2024-03-29T10:50:11.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.675+0000 7f2a3b4a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:11.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.675+0000 7f2a3b4a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:11.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.675+0000 7f2a3b4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f2a340b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T10:50:11.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a3b4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f2a341422b0 addr v1:172.21.15.114:6789/0 2024-03-29T10:50:11.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a39244640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f2a340b17d0 0x7f2a340ac870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:38060/0 (socket says 172.21.15.114:38060) 2024-03-29T10:50:11.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a38a43640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2a341422b0 0x7f2a340acf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40706/0 (socket says 172.21.15.114:40706) 2024-03-29T10:50:11.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a38a43640 1 -- 172.21.15.114:0/64382368 learned_addr learned my addr 172.21.15.114:0/64382368 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T10:50:11.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a3b4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f2a34142690 addr v1:172.21.15.176:6789/0 2024-03-29T10:50:11.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a29ffb640 10 client.?.objecter ms_handle_connect 0x7f2a341422b0 2024-03-29T10:50:11.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:11.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a29ffb640 10 client.?.objecter ms_handle_connect 0x7f2a340b17d0 2024-03-29T10:50:11.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:11.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a340b00d0 con 0x7f2a341422b0 2024-03-29T10:50:11.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a29ffb640 10 client.?.objecter ms_handle_connect 0x7f2a34142690 2024-03-29T10:50:11.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:11.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.676+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a340b09a0 con 0x7f2a34142690 2024-03-29T10:50:11.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a340b0c10 con 0x7f2a340b17d0 2024-03-29T10:50:11.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a3b4a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:11.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a3b4a8640 10 monclient(hunting): authenticate will time out at 1476.918250s 2024-03-29T10:50:11.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688802777 0 0) 0x7f2a340b00d0 con 0x7f2a341422b0 2024-03-29T10:50:11.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 10 monclient(hunting): my global_id is 4611 2024-03-29T10:50:11.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a18004eb0 con 0x7f2a341422b0 2024-03-29T10:50:11.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371662372 0 0) 0x7f2a340b0c10 con 0x7f2a340b17d0 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 10 monclient(hunting): my global_id is 4712 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.677+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a340b00d0 con 0x7f2a340b17d0 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2438192908 0 0) 0x7f2a18004eb0 con 0x7f2a341422b0 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a340b0c10 con 0x7f2a341422b0 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2a300018b0 con 0x7f2a341422b0 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 10 monclient(hunting): dump: 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:50:11.672 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 275059550 0 0) 0x7f2a340b00d0 con 0x7f2a340b17d0 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a18004eb0 con 0x7f2a340b17d0 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2a2c001c30 con 0x7f2a340b17d0 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 10 monclient(hunting): dump: 2024-03-29T10:50:11.673 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2362178269 0 0) 0x7f2a340b0c10 con 0x7f2a341422b0 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 >> v1:172.21.15.114:6790/0 conn(0x7f2a340b17d0 legacy=0x7f2a340ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 >> v1:172.21.15.176:6789/0 conn(0x7f2a34142690 legacy=0x7f2a341e1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:11.674 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 1 monclient: found mon.a 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a340b0ef0 con 0x7f2a341422b0 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a3b4a8640 5 monclient: authenticate success, global_id 4611 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a3b4a8640 10 monclient: _renew_subs 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a3b4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.678+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a340aed70 con 0x7f2a341422b0 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.679+0000 7f2a3b4a8640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.679+0000 7f2a3b4a8640 10 monclient: _renew_subs 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.679+0000 7f2a3b4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.679+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a340af9f0 con 0x7f2a341422b0 2024-03-29T10:50:11.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.679+0000 7f2a3b4a8640 1 librados: init done 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.680+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a300047e0 con 0x7f2a341422b0 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.680+0000 7f2a29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.680+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1668667138 0 0) 0x7f2a30004af0 con 0x7f2a341422b0 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.680+0000 7f2a29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.680+0000 7f2a29ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.680+0000 7f2a29ffb640 10 monclient: dump: 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:fsid aa12b535-44a2-4ddb-9a33-48fb3a93e012 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T10:38:35.262839+0000 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T10:50:11.676 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.680+0000 7f2a2b7fe640 4 set_mon_vals no callback set 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.680+0000 7f2a3b4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.681+0000 7f2a3b4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.681+0000 7f2a3b4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.681+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a14005080 con 0x7f2a341422b0 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.682+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3878350446 0 0) 0x7f2a3001aac0 con 0x7f2a341422b0 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.683+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1764351048 0 0) 0x7f2a30054820 con 0x7f2a341422b0 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.683+0000 7f2a29ffb640 10 client.4611.objecter ms_dispatch 0x7f2a34000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.683+0000 7f2a29ffb640 3 client.4611.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.683+0000 7f2a29ffb640 3 client.4611.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.683+0000 7f2a29ffb640 20 client.4611.objecter dump_active .. 0 homeless 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.684+0000 7f2a39244640 10 monclient: get_auth_request con 0x7f2a1806e050 auth_method 0 2024-03-29T10:50:11.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.684+0000 7f2a29ffb640 10 client.4611.objecter ms_handle_connect 0x7f2a1806e050 2024-03-29T10:50:11.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.692+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a30050ac0 con 0x7f2a341422b0 2024-03-29T10:50:11.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.692+0000 7f2a29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:11.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:11.692+0000 7f2a29ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:12.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.034+0000 7f2a3b4a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:12.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.034+0000 7f2a3b4a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:12.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.034+0000 7f2a3b4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T10:50:12.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.034+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a14005330 con 0x7f2a341422b0 2024-03-29T10:50:12.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.035+0000 7f2a29ffb640 1 -- 172.21.15.114:0/64382368 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2201894833) 0x7f2a30050c60 con 0x7f2a341422b0 2024-03-29T10:50:12.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.035+0000 7f2a29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:12.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.035+0000 7f2a29ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:12.017 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T10:50:12.017 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"aa12b535-44a2-4ddb-9a33-48fb3a93e012","created":"2024-03-29T10:39:13.991730+0000","modified":"2024-03-29T10:49:18.930439+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:18.930439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b22e9cf8-6612-4468-b6b2-a6c1574f2921","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d8ca84b0-980e-461e-962f-0bcb7e2cf005","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d83b07bd-dc3c-4266-b151-9bd1e1bf252d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"94c64ca0-9b70-4f5a-bb1c-5daebf7ad29d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1585ada0-eaef-41fc-b41d-179f7c1756b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7aa69521-7d4d-4c15-8e3e-5f1dfceafcb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9230491b-7e8a-4789-8f62-c189a38c9ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1164e467-12f7-40e9-9770-7e27c30cfa84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:12.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.038+0000 7f2a3b4a8640 10 librados: watch_flush enter 2024-03-29T10:50:12.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.038+0000 7f2a3b4a8640 10 librados: watch_flush exit 2024-03-29T10:50:12.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.038+0000 7f2a3b4a8640 20 client.4611.objecter shutdown clearing up homeless session... 2024-03-29T10:50:12.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.038+0000 7f2a3b4a8640 10 client.4611.objecter successfully canceled tick 2024-03-29T10:50:12.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.038+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 >> v1:172.21.15.114:6816/775044307 conn(0x7f2a1806e050 legacy=0x7f2a18070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:12.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.038+0000 7f2a3b4a8640 10 monclient: shutdown 2024-03-29T10:50:12.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.038+0000 7f2a3b4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:12.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.038+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 >> v1:172.21.15.114:6789/0 conn(0x7f2a341422b0 legacy=0x7f2a340acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:12.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.038+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 shutdown_connections 2024-03-29T10:50:12.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.038+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 >> 172.21.15.114:0/64382368 conn(0x7f2a34066a00 msgr2=0x7f2a340b3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:12.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.038+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 shutdown_connections 2024-03-29T10:50:12.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.039+0000 7f2a3b4a8640 1 -- 172.21.15.114:0/64382368 wait complete. 2024-03-29T10:50:12.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T10:50:12.039+0000 7f2a3b4a8640 1 librados: shutdown 2024-03-29T10:50:12.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:12.035 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:50:12.036 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T10:50:12.036 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T10:50:12.037 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.069 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T10:50:12.070 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T10:50:12.070 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.144 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T10:50:12.144 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T10:50:12.145 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.216 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T10:50:12.216 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T10:50:12.216 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.287 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T10:50:12.287 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T10:50:12.287 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.321 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T10:50:12.321 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T10:50:12.321 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.393 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T10:50:12.393 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T10:50:12.393 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.463 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T10:50:12.463 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T10:50:12.463 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.529 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T10:50:12.530 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T10:50:12.530 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T10:50:12.530 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.555 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T10:50:12.555 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T10:50:12.555 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.594 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T10:50:12.594 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T10:50:12.595 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T10:50:12.595 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.606 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T10:50:12.607 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T10:50:12.607 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.659 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T10:50:12.659 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T10:50:12.659 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:12.670 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T10:50:12.670 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T10:50:12.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T10:50:12.730 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T10:50:12.731 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T10:50:15.675 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T10:50:15.675 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T10:50:15.757 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T10:50:15.757 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T10:50:15.823 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T10:50:15.823 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T10:50:15.891 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T10:50:15.891 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T10:50:19.405 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T10:50:19.405 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T10:50:19.474 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T10:50:19.475 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T10:50:19.548 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T10:50:19.548 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T10:50:19.599 INFO:tasks.ceph:Archiving mon data... 2024-03-29T10:50:19.601 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645/data/mon.a.tgz 2024-03-29T10:50:19.601 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-03-29T10:50:19.619 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.sr9WCS3VuD 2024-03-29T10:50:19.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.sr9WCS3VuD 2024-03-29T10:50:19.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.sr9WCS3VuD 2024-03-29T10:50:19.903 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.sr9WCS3VuD is 211KB 2024-03-29T10:50:19.997 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.sr9WCS3VuD 2024-03-29T10:50:20.013 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645/data/mon.c.tgz 2024-03-29T10:50:20.013 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-03-29T10:50:20.068 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.uiIDqLr8d9 2024-03-29T10:50:20.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.uiIDqLr8d9 2024-03-29T10:50:20.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.uiIDqLr8d9 2024-03-29T10:50:20.253 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.uiIDqLr8d9 is 211KB 2024-03-29T10:50:20.346 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.uiIDqLr8d9 2024-03-29T10:50:20.361 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645/data/mon.b.tgz 2024-03-29T10:50:20.361 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-03-29T10:50:20.379 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.hIOUgABv8s 2024-03-29T10:50:20.379 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.hIOUgABv8s 2024-03-29T10:50:20.500 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.hIOUgABv8s 2024-03-29T10:50:20.589 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.hIOUgABv8s is 211KB 2024-03-29T10:50:20.682 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.hIOUgABv8s 2024-03-29T10:50:20.697 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T10:50:20.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:50:20.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:50:20.781 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:50:20.820 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:50:20.820 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T10:50:20.822 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645/remote/smithi114/crash 2024-03-29T10:50:20.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:50:20.850 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645/remote/smithi176/crash 2024-03-29T10:50:20.851 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:50:20.887 INFO:tasks.ceph:Compressing logs... 2024-03-29T10:50:20.888 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:50:20.895 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:50:20.919 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T10:50:20.919 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39137.log 2024-03-29T10:50:20.919 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40229.log 2024-03-29T10:50:20.919 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.39137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38981.log 2024-03-29T10:50:20.958 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40307.log 2024-03-29T10:50:20.958 INFO:teuthology.orchestra.run.smithi114.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39137.log.gz 2024-03-29T10:50:20.958 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38981.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40229.log.gz 2024-03-29T10:50:20.958 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41651.log 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40307.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38981.log.gz 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41339.log 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41222.log 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40307.log.gz 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39566.log 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41651.log.gz 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38747.log 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41339.log.gz 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40754.log 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41222.log.gz 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38747.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39566.log.gz 2024-03-29T10:50:20.959 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39722.log 2024-03-29T10:50:20.960 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39917.log 2024-03-29T10:50:20.960 INFO:teuthology.orchestra.run.smithi114.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38747.log.gz 2024-03-29T10:50:20.960 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40385.log 2024-03-29T10:50:20.960 INFO:teuthology.orchestra.run.smithi114.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40754.log.gz 2024-03-29T10:50:20.960 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39878.log 2024-03-29T10:50:20.960 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39722.log.gz 2024-03-29T10:50:20.960 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40385.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39917.log.gz 2024-03-29T10:50:20.960 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40871.log 2024-03-29T10:50:20.960 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41807.log 2024-03-29T10:50:20.960 INFO:teuthology.orchestra.run.smithi114.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40385.log.gz 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40871.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39878.log.gz 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41807.log: gzip 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40871.log.gz -5 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39605.log 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39254.log 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41807.log.gz 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39605.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40424.log 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39332.log 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39605.log.gz 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38786.log 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39254.log.gz 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39332.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40424.log.gz 2024-03-29T10:50:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41612.log 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39683.log 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39332.log.gz 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41066.log 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38786.log.gz 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39683.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41612.log.gz 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41690.log 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40346.log 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39683.log.gz 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39449.log 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41066.log.gz 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40346.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41690.log.gz 2024-03-29T10:50:20.962 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38548.log 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39761.log 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39449.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40346.log.gz 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38548.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39761.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39449.log.gz 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38825.log 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38548.log.gz 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr:gzip/var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39410.log 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39761.log.gz 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38942.log 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39410.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38825.log.gz 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38626.log 2024-03-29T10:50:20.963 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40637.log 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39410.log.gz 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38903.log 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38942.log.gz 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41417.log 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38626.log.gz 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38469.log 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40637.log.gz 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41417.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38903.log.gz 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39293.log 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39800.log 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38469.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41417.log.gz 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39371.log 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38469.log.gz 2024-03-29T10:50:20.964 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41378.log 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39293.log.gz 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35093.log 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39800.log.gz 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40034.log 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39371.log.gz 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40832.log 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41378.log.gz 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39644.log 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35093.log.gz 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40034.log.gz 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40832.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39020.log 2024-03-29T10:50:20.965 INFO:teuthology.orchestra.run.smithi114.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40832.log.gz 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41261.log 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39644.log.gz 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40559.log 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41261.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39020.log.gz 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41456.log 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41261.log.gz 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38708.log 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40559.log: /var/log/ceph/ceph-client.admin.41456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40598.log 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr: 86.1%/var/log/ceph/ceph-client.admin.38708.log: -- replaced with /var/log/ceph/ceph-client.admin.40559.log.gz 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40112.log 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41456.log.gz 2024-03-29T10:50:20.966 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38708.log.gz 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39956.log 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40598.log.gz 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr:gzip/var/log/ceph/ceph-client.admin.40676.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39098.log 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40112.log.gz 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40715.log 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41573.log 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39956.log.gz 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40715.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39098.log.gz 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41144.log 2024-03-29T10:50:20.968 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40715.log.gz 2024-03-29T10:50:20.968 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39488.log 2024-03-29T10:50:20.968 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41573.log.gz 2024-03-29T10:50:20.968 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40190.log 2024-03-29T10:50:20.968 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41846.log 2024-03-29T10:50:20.968 INFO:teuthology.orchestra.run.smithi114.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41144.log.gz 2024-03-29T10:50:20.968 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34990.log 2024-03-29T10:50:20.968 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39488.log.gz 2024-03-29T10:50:20.968 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41846.log: -- /var/log/ceph/ceph-client.admin.41534.log 2024-03-29T10:50:20.968 INFO:teuthology.orchestra.run.smithi114.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40190.log.gz 2024-03-29T10:50:20.968 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40268.log 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41846.log.gz 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41534.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34990.log.gz 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40467.log 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40268.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41534.log.gz 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39059.log 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40268.log.gz 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40151.log 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40988.log 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39059.log: /var/log/ceph/ceph-client.admin.40151.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40467.log.gz 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41729.log 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39059.log.gz 2024-03-29T10:50:20.969 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40988.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40151.log.gz 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41027.log 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr:gzip/var/log/ceph/ceph-client.admin.41729.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40073.log 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40988.log.gz 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38390.log 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41729.log.gz 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40910.log 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41027.log.gz 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38390.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40073.log.gz 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34700.log 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38390.log.gz 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28880.log 2024-03-29T10:50:20.970 INFO:teuthology.orchestra.run.smithi114.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40910.log.gz 2024-03-29T10:50:20.971 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-03-29T10:50:20.971 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.tmp-client.admin.28880.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28880.log.gz 2024-03-29T10:50:20.971 INFO:teuthology.orchestra.run.smithi114.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39995.log 2024-03-29T10:50:20.971 INFO:teuthology.orchestra.run.smithi114.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34700.log.gz 2024-03-29T10:50:20.971 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40949.log 2024-03-29T10:50:20.971 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39995.log: 86.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40518.log 2024-03-29T10:50:20.971 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-03-29T10:50:20.971 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40949.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T10:50:20.971 INFO:teuthology.orchestra.run.smithi114.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39995.log.gz 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38667.log 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40518.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40949.log.gz 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5/var/log/ceph/ceph-osd.0.log: --verbose -- /var/log/ceph/ceph-client.admin.38864.log 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38667.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40518.log.gz 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39527.log 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38667.log.gz 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38864.log: /var/log/ceph/ceph-client.admin.39527.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38864.log.gz 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39527.log.gz 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40793.log 2024-03-29T10:50:20.972 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38429.log 2024-03-29T10:50:20.973 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40793.log: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:50:20.973 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40793.log.gz 2024-03-29T10:50:20.973 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38509.log 2024-03-29T10:50:20.973 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39176.log 2024-03-29T10:50:20.973 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38429.log: /var/log/ceph/ceph-client.admin.38509.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38429.log.gz 2024-03-29T10:50:20.973 INFO:teuthology.orchestra.run.smithi114.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38509.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41300.log 2024-03-29T10:50:20.973 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:50:20.973 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39176.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39176.log.gz 2024-03-29T10:50:20.973 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39839.log 2024-03-29T10:50:20.973 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39215.log 2024-03-29T10:50:20.974 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41300.log: /var/log/ceph/ceph-client.admin.39839.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41300.log.gz 2024-03-29T10:50:20.974 INFO:teuthology.orchestra.run.smithi114.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39839.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41495.log 2024-03-29T10:50:20.974 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:50:20.974 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39215.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39215.log.gz 2024-03-29T10:50:20.974 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38587.log 2024-03-29T10:50:20.974 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41768.log 2024-03-29T10:50:20.974 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41495.log: /var/log/ceph/ceph-client.admin.38587.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41495.log.gz 2024-03-29T10:50:20.974 INFO:teuthology.orchestra.run.smithi114.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38587.log.gz 2024-03-29T10:50:20.974 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T10:50:20.974 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41768.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41768.log.gz 2024-03-29T10:50:20.983 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T10:50:20.983 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:50:20.983 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34621.log 2024-03-29T10:50:20.983 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34461.log 2024-03-29T10:50:20.983 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34621.log: 88.4%gzip -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:50:20.983 INFO:teuthology.orchestra.run.smithi176.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T10:50:20.983 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34421.log 2024-03-29T10:50:20.983 INFO:teuthology.orchestra.run.smithi176.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34621.log.gz 2024-03-29T10:50:20.983 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34461.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34701.log 2024-03-29T10:50:20.983 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34421.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34461.log.gz 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34701.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34421.log.gz 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34661.log 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.7.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34701.log.gz 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34661.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34581.log 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr: 87.8%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.34661.log.gz 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34501.log 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34581.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34581.log.gz 2024-03-29T10:50:20.984 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34541.log 2024-03-29T10:50:20.985 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mon.b.log: 87.8%/var/log/ceph/ceph-client.admin.34541.log: -- replaced with /var/log/ceph/ceph-client.admin.34501.log.gz 2024-03-29T10:50:20.985 INFO:teuthology.orchestra.run.smithi176.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34541.log.gz 2024-03-29T10:50:20.985 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T10:50:21.012 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mon.c.log: 92.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T10:50:32.952 INFO:teuthology.orchestra.run.smithi176.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T10:50:33.051 INFO:teuthology.orchestra.run.smithi114.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T10:50:33.099 INFO:teuthology.orchestra.run.smithi114.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T10:50:33.115 INFO:teuthology.orchestra.run.smithi176.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T10:50:33.155 INFO:teuthology.orchestra.run.smithi176.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T10:50:37.226 INFO:teuthology.orchestra.run.smithi176.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T10:50:37.254 INFO:teuthology.orchestra.run.smithi114.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T10:50:37.298 INFO:teuthology.orchestra.run.smithi114.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T10:51:13.388 INFO:teuthology.orchestra.run.smithi114.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T10:51:18.550 INFO:teuthology.orchestra.run.smithi114.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T10:51:18.552 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T10:51:18.552 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m57.646s 2024-03-29T10:51:18.553 INFO:teuthology.orchestra.run.smithi114.stderr:user 2m41.128s 2024-03-29T10:51:18.553 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m5.746s 2024-03-29T10:51:31.094 INFO:teuthology.orchestra.run.smithi176.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T10:51:31.095 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T10:51:31.095 INFO:teuthology.orchestra.run.smithi176.stderr:real 1m10.150s 2024-03-29T10:51:31.095 INFO:teuthology.orchestra.run.smithi176.stderr:user 1m58.129s 2024-03-29T10:51:31.096 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m4.410s 2024-03-29T10:51:31.096 INFO:tasks.ceph:Archiving logs... 2024-03-29T10:51:31.096 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645/remote/smithi114/log 2024-03-29T10:51:31.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:51:56.078 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645/remote/smithi176/log 2024-03-29T10:51:56.079 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:52:14.415 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:52:14.637 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=203f447e26ba43ddb806ac7b1f2c6340 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:52:14.641 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T10:52:14.649 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T10:52:14.668 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T10:52:14.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:52:14.676 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:52:14.758 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T10:52:14.766 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T10:52:14.767 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:52:14.769 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:52:14.783 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:52:14.785 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:52:14.787 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:52:14.787 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-03-29T10:52:14.787 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 +431us[ +431us] +/- 70ms 2024-03-29T10:52:14.787 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -189us[ -189us] +/- 50ms 2024-03-29T10:52:14.787 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 -204us[ -204us] +/- 81ms 2024-03-29T10:52:14.787 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:52:14.790 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:52:14.790 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-03-29T10:52:14.790 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +430us[ +436us] +/- 70ms 2024-03-29T10:52:14.790 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -170us[ -164us] +/- 50ms 2024-03-29T10:52:14.790 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 -268us[ -268us] +/- 81ms 2024-03-29T10:52:14.791 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:52:14.791 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T10:52:14.801 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T10:52:14.802 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T10:52:14.827 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:52:14.897 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:52:14.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:52:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:52:14.999 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:52:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:52:15.031 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T10:52:15.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T10:52:15.047 INFO:teuthology.task.internal:Duration was 1274.790489 seconds 2024-03-29T10:52:15.048 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T10:52:15.056 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T10:52:15.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:52:15.058 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:52:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:52:15.116 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:52:15.450 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T10:52:15.450 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T10:52:15.451 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:52:15.488 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T10:52:15.488 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:52:15.515 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T10:52:15.515 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:52:15.531 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:52:15.584 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T10:52:15.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:52:15.600 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:52:15.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T10:52:15.731 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T10:52:15.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:52:15.758 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:52:15.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T10:52:15.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:52:15.801 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:52:15.824 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-03-29T10:52:15.850 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-03-29T10:52:15.885 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:52:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:52:15.904 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:52:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:52:15.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T10:52:15.950 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T10:52:15.952 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645/remote/smithi114 2024-03-29T10:52:15.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:52:16.082 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645/remote/smithi176 2024-03-29T10:52:16.082 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:52:16.124 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T10:52:16.125 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:52:16.128 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:52:16.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T10:52:16.182 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T10:52:16.182 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T10:52:16.194 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T10:52:16.194 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:52:16.201 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:52:16.213 INFO:teuthology.orchestra.run.smithi114.stdout: 265661 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:52 /home/ubuntu/cephtest 2024-03-29T10:52:16.228 INFO:teuthology.orchestra.run.smithi176.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:52 /home/ubuntu/cephtest 2024-03-29T10:52:16.229 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T10:52:16.247 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T10:52:16.286 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T10:52:16.320 DEBUG:teuthology.nuke:shortname: smithi114 2024-03-29T10:52:16.321 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:52:16.349 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:22:52.591831', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDR2+Wh4EFdavoNAlwXZOvp3kKLwaLJcfAlicDL6P656byf9my2aWpeW5A3gmSLnaGRIoRC2nm50uiRzDUcvhsI='} 2024-03-29T10:52:16.354 DEBUG:teuthology.nuke:shortname: smithi176 2024-03-29T10:52:16.354 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:52:16.378 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622645', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:22:52.593074', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApCkfzolq0sEtzqzhFSLQvasiZ2FY7OxW3B1Ce2+TbmTeFff2PlE1cXJlwWU7g+i40pHhoavGZC0srYa6M9OlY='} 2024-03-29T10:52:16.414 INFO:teuthology.orchestra.console.smithi114:Power off 2024-03-29T10:52:16.415 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:52:16.470 INFO:teuthology.orchestra.console.smithi176:Power off 2024-03-29T10:52:16.470 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:52:16.483 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-03-29T10:52:16.484 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:16.502 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-03-29T10:52:16.502 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:16.515 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-29T10:52:16.532 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-29T10:52:20.517 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:20.533 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:20.630 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-29T10:52:20.646 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-29T10:52:24.634 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:24.648 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:24.746 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-29T10:52:24.757 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-03-29T10:52:24.858 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-03-29T10:52:28.750 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:28.860 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-03-29T10:52:28.961 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-03-29T10:52:29.065 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1274.7904891967773 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=203f447e26ba43ddb806ac7b1f2c6340 status: fail success: false 2024-03-29T10:52:29.066 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:52:29.145 INFO:teuthology.run:FAIL